[help] Constructing a synced sequence input and output RNN

Hi there,

I’m building an RNN to assign an output label for each input element in the sequence for activity recognition based on location. In this toy model, the shape of each input location is 4x1; the shape of each output activity is 3x1. There are two hidden layers, the shape of each hidden component is 3x1.

My question is how to construct the model? Do I need to use Embedding layer? Should I use two layers of TimeDistributedDense or two layers of GRU/LSTM for my two hidden layers?

Please help and I hope I could contribute an example to the repo 😃

My code snippet is shown below.

input_dim = 4
output_dim = 3
hidden_dim = 3

print('Build model...')

model = Sequential()

# TODO: add layers to model

print('Compile model...')
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
model.fit(X_train, Y_train, batch_size=1, nb_epoch=10)
print('Done')

[patyork]

No, TimeDistrubutedDense is exactly as it sounds, simply a Dense layer that feed all of its inputs forward in time; this distinction between Dense and TimeDistributedDense is simply that a Dense layer expects 2D input (batch_size, sample_size) whereas TimeDistributedDense expects 3D input (Batch_size, time_steps, sample_size). This should be used in conjunction with TimeDistributedSoftmax for the same reason (2D vs. 3D expected input).

There is a GRU layer, however: https://github.com/fchollet/keras/blob/master/keras/layers/recurrent.py#L156-253

[lemuriandezapada]

1. yes, that is the correct way. (except that your activities don’t sum up to 1, nor are they capped at one, just use a linear layer instead)
2. TimeDistributedDense is like Dense (fully connected) but spread over time. (not connected on the time dimension) 😃)
3. The problem you’ll have isn’t so much the dropout as it is the initialization. Usually these LSTMs won’t be stackable for more than 2 layers just initialized at random. It’s quite likely your network won’t learn anything. You need to do some kind of pretraining first. I often find that doing it with one recurrent layer first for a few hundred epochs then removing the top layer, adding another random recurrent layer on top of that and then resuming works decently.