How to predict Time Series?

Hello, I’ve been playing around with the example for a bit and cannot solve one issue. I want to predict a time series, for now a simple cosine function. The function is irregularly sampled at 250 points. How I have been doing this so far is to chunk the time series into chunks of length 50, and fit a spline on each chunk with their corresponding time stamps scaled to [0,1]. I set the times for the model to a linspace(0, 1, 50). This does not work. How should I go about setting everything up properly? Here is my code for reference (I hope it is not too much of a mess). (Only the relevent part from the example. Additionally, I changed the solver to dopri8, available in the newest version of torchdiffeq. It gave better results than all the other solvers at the cost of being slightly slower)

def get_data():
    # some time series I had at hand
    d = np.load('/XXX/tsdata/300.npy')
    
    # the timestamps
    t = d[0]
    
    # generating the cos function
    Xfull = np.cos(t*10*np.pi)
    Xfull = (Xfull - np.mean(Xfull)) / np.std(Xfull)
    
    ts = []
    Xs = []
    ys = []
    
    tstest = []
    Xstest = []
    ystest = []
    
    # splitting the dataset into train and test
    for i in range(0, len(Xfull)-101):
        ts.append(t[i:i+50])
        ts[-1] = (ts[-1] - np.min(ts[-1])) / (np.max(ts[-1]) - np.min(ts[-1]))
        Xs.append(Xfull[i:i+50])
        ys.append(Xfull[i+50])

    for i in range(len(Xfull)-101, len(Xfull)-51):
        tstest.append(t[i:i+50])
        tstest[-1] = (tstest[-1] - np.min(tstest[-1])) / (np.max(tstest[-1]) - np.min(tstest[-1]))
        Xstest.append(Xfull[i:i+50])
        ystest.append(Xfull[i+50])
        
    return (torch.Tensor(ts), torch.Tensor(Xs), torch.Tensor(ys),
            torch.Tensor(tstest), torch.Tensor(Xstest), torch.Tensor(ystest))



def main(): 
    from collections import defaultdict
    train_t, train_X, train_y, test_t, test_X, test_y = get_data()


    device = torch.device('cuda')
    model = NeuralCDE(input_channels=1, hidden_channels=12, output_channels=1).to(device)
    optimizer = torch.optim.Adam(model.parameters())
    
    # getting the spline coefficients
    # done like this because the timestamps are different for each chunk
    train_coeffs = []
    for t, X in zip(train_t, train_X):
        train_coeffs.append(controldiffeq.natural_cubic_spline_coeffs(t, torch.unsqueeze(X, 0).unsqueeze(-1)))
    elements = defaultdict(list)

    final_coeffs = []
    for c in train_coeffs:
        for ix, el in enumerate(c):
            elements[ix].append(el)

    for i in elements.keys():
        final_coeffs.append(torch.stack(elements[i]).squeeze(1))

    train_coeffs = tuple(final_coeffs)
    train_y = train_y.to(device)
    
    # getting the spline coefficients for the test data
    test_coeffs = []
    for t, X in zip(test_t, test_X):
        test_coeffs.append(controldiffeq.natural_cubic_spline_coeffs(t, torch.unsqueeze(X, 0).unsqueeze(-1)))
    elements = defaultdict(list)

    final_coeffs = []
    for c in test_coeffs:
        for ix, el in enumerate(c):
            elements[ix].append(el)

    for i in elements.keys():
        final_coeffs.append(torch.stack(elements[i]).squeeze(1))

    test_coeffs = tuple(final_coeffs)
    test_y = test_y.to(device)
    
    train_dataset = torch.utils.data.TensorDataset(*train_coeffs, train_y)
    train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=256, shuffle=True)

    test_dataset = torch.utils.data.TensorDataset(*test_coeffs, test_y)
    test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=256, shuffle=False)

    criterion = torch.nn.MSELoss()

    ts = torch.Tensor(np.linspace(0, 1, train_t.shape[1])).to(device)

    test_predictions = []

    for epoch in range(500):
        for batch in train_dataloader:
            *batch_coeffs, batch_y = batch
            coeffs = []
            for c in batch_coeffs:
                coeffs.append(c.to(device))
            pred_y = model(ts, coeffs).squeeze(-1)
            loss = criterion(pred_y, batch_y)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

        model.eval()
        with torch.no_grad():
            test_predictions = [[], []]
            for batch in test_dataloader:
                *batch_coeffs, batch_y = batch
                coeffs = []
                for c in batch_coeffs:
                    coeffs.append(c.to(device))
                pred_y = model(ts, coeffs).squeeze(-1)
                test_predictions[0].append(pred_y.detach().cpu().numpy())
                test_predictions[1].append(batch_y.detach().cpu().numpy())
        model.train()
        
        np.save(f"/home/XXX/preds/{epoch}_p.npy", np.array(test_predictions))
            
        print('Epoch: {}   Training loss: {}'.format(epoch, loss.item()))

[patrick-kidger]

So the issue here is that you only have one training sample. The usual pattern in supervised learning is to have lots of training samples, and train the model on the whole collection. Neural CDEs aren’t any different here - one way or another you need to try and get more training data. (Which can be by taking your one sample and cutting it up into pieces as you did originally - I now understand why you were doing that.)

In terms of forecasting forwards, I’d suggesting using a sequence-to-sequence architecture, with a Neural CDE as the encoder, and for example a Neural ODE as the decoder. This naturally captures the structure of the problem.

Lastly - and I realise this isn’t done in the example, which is something I should fix (EDIT: now fixed) - I’d strongly recommend appending time as a channel to your input. So you have a two-dimensional input; one channel is time and one channel are your observations. (This is for complicated mathematics reasons: CDEs don’t notice the speed at which you pass data unless you explicitly set them to do so; it’s called the reparameterisation invariance property).

[he-ritter]

Thanks for the suggestions! After implementing your suggestions, things started working out. I tried a NeuralODE decoder for Seq2Seq, the results were actually worse than just using the NeuralCDE model. Can you maybe say what a resonable approach to set the initial z0 would be? In the example, it is set to all 0s, but I got better results when I tried to set it to e.g. an encoding of the first X values of the input time series. Else it would be impossible for the CDE to capture trends in the time series correctly, if I understand correctly. (As z0 + integral(f(z) * dX/dt) cannot be moved up/down if z0 is always 0) I understand that the example is only set up for the particular problem presented there, and that the approach taken there is probably not the ideal one for every use case. But so far, this package is doing wonders to time series prediction with a low number of samples. Other models never worked, this is finding a reasonable solution quite quickly. Impressive!