Integrated gradient for LSTM shows 'cudnn RNN backward can only be called in training mode'

Hi all,

I have a trained LSTM model and the output is a time series. I tried to apply integrated gradient but got the error showing ‘cudnn RNN backward can only be called in training mode’. Any ideas how to solve it? Here is the code.

def forward_fun(x_d, x_s, x_one_hot):
    out = lstm(x_d, x_s, x_one_hot)[0][:,-1]  #0 is for the cell output and -1 is for the last index in the ouput series
    return out
ig = IntegratedGradients(forward_fun) 

attrs = []
for i in range(x_d.shape[0]):
    attr = ig.attribute(inputs=(x_d[i:i+1]), additional_forward_args=(x_s[i:i+1], x_one_hot)).cpu().data.numpy()
    attrs.append(attr)
attrs = np.concatenate(attrs, axis=0)

Then I tried to move all the model and data to CPU and it is working now (slowly). How can I make it on the GPU??

Thanks all!

[NarineK]

@ShihengDuan , I saw similar problem too and setting the following flag helped to fix the problem:

torch.backends.cudnn.enabled=False

[NarineK]

@ShihengDuan, CuDNN with RNN doesn’t support gradient computation in eval mode that’s why we need to disable cudnn for RNN in eval mode. I do not know the exact reasons. That would be a good question for pytorch discussion forum.