Input and target format for multidimentional time-series regression

I’m trying to solve a problem I had intended for tensorflow in Keras.

I’ve gotten a lot further using Keras, but I’m still unclear on how best to represent my sequence data. The following code works quite well using only one input sample and one target sample:

import numpy as np

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM

# This does work by using only one sample:
data = [[0,0,0,0,0,0,0,0,0,2,1]]
data = np.array(data, dtype=float)
target = [0,0,0,0,0,0,0,0,2,1,0]
target = np.array(target, dtype=float)

data = data.reshape((1, 1, 11)) # Single batch, 1 time steps, 11 dimentions
target = target.reshape((-1, 11)) # Corresponds to shape (None, 11)


# Build Model
model = Sequential()  
model.add(LSTM(11, input_shape=(1, 11), unroll=True))
model.add(Dense(11))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(data, target, nb_epoch=1000, batch_size=1, verbose=2)

# Do the output values match the target values?
predict = model.predict(data)
print repr(data)
print repr(predict)

But I can’t get it to work with multiple samples:

import numpy as np

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM

# Input sequence
wholeSequence = [[0,0,0,0,0,0,0,0,0,2,1],
                 [0,0,0,0,0,0,0,0,2,1,0],
                 [0,0,0,0,0,0,0,2,1,0,0],
                 [0,0,0,0,0,0,2,1,0,0,0],
                 [0,0,0,0,0,2,1,0,0,0,0],
                 [0,0,0,0,2,1,0,0,0,0,0],
                 [0,0,0,2,1,0,0,0,0,0,0],
                 [0,0,2,1,0,0,0,0,0,0,0],
                 [0,2,1,0,0,0,0,0,0,0,0],
                 [2,1,0,0,0,0,0,0,0,0,0]]

# Preprocess Data: (This does not work)
wholeSequence = np.array(wholeSequence, dtype=float) # Convert to NP array.
data = wholeSequence[:-1] # all but last
target = wholeSequence[1:] # all but first

# This does not work:
# Reshape training data for Keras LSTM model
# The training data needs to be (batchIndex, timeStepIndex, dimentionIndex)
data = data.reshape((1, 9, 11)) # Single batch, 9 time steps, 11 dimentions
target = target.reshape((-1, 11)) # Corresponds to shape (None, 11)


# Build Model
model = Sequential()  
model.add(LSTM(11, input_shape=(9, 11), unroll=True))
model.add(Dense(11))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(data, target, nb_epoch=1000, batch_size=1, verbose=2)

# Do the output values match the target values?
predict = model.predict(data)
print repr(data)
print repr(predict)

Due to this error: ValueError: Input arrays should have the same number of samples as target arrays. Found 1 input samples and 9 target samples.

What am I doing wrong with array shape?

It would be really nice if Keras facilitates this use case such that a single data structure holds the sequence and the fitter would know that for each input X_t, the target is X_(t+1). This would provide some benefits such as the following:

• There would be no redundancy in storing the data and targets separately.

• One would not have to be concerned with the shape of the input and targets separately.

[bstriner]

The first dimension of your data is the batch dimension. It will show up as None. It can be any size, as long as it is the same for your inputs and targets. When you’re dealing with LSTMs, the batch dimension is the number of sequences, not the length of the sequence.

LSTMs in Keras are typically used on 3d data (batch dimension, timesteps, features). LSTM without return_sequences will output (batch dimension, output features) LSTM with return_sequences will output (batch dimension, timesteps, output features)

So, if your input shape is (None, 9, 11) and your actual input shape is (1, 9, 11) that means your batch dimension is 1. If your output shape is (None, 11), then your actual targets need to be (1,11).

The loop you’re describing isn’t the way to do things in Keras. Run a single vector with shape (number of sequences, steps, features) to calculate the entire series in one go. That way errors can backpropagate through time.

Are you trying to do time-series prediction? I’m not sure what you’re trying to build.

Basic timeseries data has an input shape (number of sequences, steps, features). Target is (number of sequences, steps, targets). Use an LSTM with return_sequences.

Probably skip the dense layer for now until you have the basics working.

Cheers, Ben

[bstriner]

@ss32 dimensions are (samples, timesteps, features). “Samples” are data that goes together so it doesn’t make any sense when you say you have 2 input samples and 1 output sample. Maybe you are saying you have 1 input sample (with 44 features for each timestep). Maybe you are saying you have 2 input samples and 2 output samples, but the 2 output samples are the same.

The data you’re describing is impossible, so please explain what you are trying to do with 2 input samples and 1 output sample. What is your data supposed to represent?

Number of input and output samples have to be the same because a “sample” is an input and output combination.

You might also just be confusing samples and features. 2 input features and 1 output feature is just input (1,2000,2) output (1,2000,1).

On a related note, please do not try to pass a 2000 length sequence to your LSTM. It will give you junk (unless you’re just looking to make an art project or something). Best strategy is to slice out many subsequences of maybe length 40 (something bigger than your expected time lag) so your shapes are (None, 40, 2) and (None, 40, 1). If you have very many possible different subsequences you will learn a model that works marginalized over all possible positions in the sequence. LSTM will get slow after maybe 10 steps, so 2000 is kind of silly, especially if you don’t think you have relationships with 1999 lag. Also, if you only have 1 sample, nothing will generalize. You can break that 1 sample into 1960 subsequences of length 40 and you might learn something more meaningful.

Cheers