Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

slow training of LSTM

See original GitHub issue

Hi, I just started using keras. Awesome work! I tried to use LSTM with the following code

model = Sequential()
model.add(LSTM(4096, 512, return_sequences=True))                                                                                                                
model.add(TimeDistributedDense(512, 4096))
model.add(Activation('time_distributed_softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

And I compared the efficiency with char-rnn, and found that the implementation in keras is about 4 times slower than Karpathy’s (with the same batchsize). Am I doing something wrong? I’ve attached the theano profile result Thanks you!

Issue Analytics

State:
Created 8 years ago
Comments:16 (9 by maintainers)

Top GitHub Comments

2reactions

fcholletcommented, Jul 20, 2015

Training time is heavily dependent on network size and batch size. Is it even the same network (size included) at all?

Also, time_distributed_softmax is deprecated now, use softmax instead.

0reactions

kylemcdonaldcommented, Jul 25, 2015

aha, great! i read through #98 hoping to be able to help, but it seems that implementing stateful training & prediction requires a deeper understanding of theano than i have right now. so i’ll step back and watch from the sidelines 😃