DistributedDataParallel only works with volatile gradients.

Hi, I tried to train a model using multi-gpus. The code works fine on single gpu but when i try to use multiple gpus then i get the following error:

Traceback (most recent call last):
  File "train.py", line 302, in <module>
    loss.backward()
  File "/export/gpudata/wangbo/conda/envs/py36/lib/python3.6/site-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/export/gpudata/wangbo/conda/envs/py36/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
  File "/export/gpudata/wangbo/conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 215, in distributed_data_parallel_hook
    raise RuntimeError("DistributedDataParallel only works with volatile gradients")
RuntimeError: DistributedDataParallel only works with volatile gradients
terminate called after throwing an instance of 'gloo::EnforceNotMet'

It seems that these model parameters after computed with CTCLoss function became non-volatile. It’s reasonless because the code works fine on single gpu. I thought there are something wrong with DistributedDataParallel parameter setting? I am using python3.6 with pytorch 0.3.0 and with cuda8.0 and cudnn7.0. Could you tell how did you run this multi-gpus training?

[simonwang517]

@Diamondfan Bravo, it works

[Diamondfan]

Problems may be here.

Try this: ctx.grads = Variable(grads, volatile=True)

Cause using DDP in pytorch0.3.1, it requires the grads to be volatile. Here