ConvLSTM2D with Theano libgpuarray and cuDNN crashes

KeyError: (‘The following error happened while compiling the node’, forall_inplace,cpu,scan_fn}(Shape_i{1}.0, InplaceGpuDimShuffle{0,1,4,2,3}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, Shape_i{1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0), ‘\n’, ‘The following error happened while compiling the node’, GpuDnnConv{algo=‘small’, inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype=‘float32’, context_name=None}.0, GpuDnnConvDesc{border_mode=‘half’, subsample=(1, 1), conv_mode=‘conv’, precision=‘float32’}.0, Constant{1.0}, Constant{0.0}), ‘\n’, ‘cudnn_handle’)

I assume this is related to having convolution operations inside a scan. @jeammimi, @nouiz, @fchollet, what do you think? Can you get the conv_lstm.py example working with THEANO_FLAGS=device=cuda and cuDNN?

[kilotaras]

The fix is in #4458

[carlthome]

Solved!

I’m on Ubuntu 16.04. For CUDA 7.5, if you recall, the gcc version had to be no newer than 4.9, so many of us used update-alternatives to lock versions. Now, with CUDA 8 nvcc wants a newer gcc so the solution was to configure update-alternatives properly.

One would also have to clear Theano’s compile cache most likely (theano-cache purge).