Error if the gradient of tensor is None.

The gradient of the tensor may be None, if the tensor is forward, but do not backward.

For example, I’m using BERT to finetune a model with the last second enocded_layer. The last layer is calculated when forward, however, it’s not gradient do not be calculated when backward.

The following is the error message.

File "/usr/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/optimizers/fp16_optimizer.py", line 147, in step
    grads_groups_flat.append(_flatten_dense_tensors([p.grad for p in group]))
File "/usr/lib/python3.7/site-packages/torch/_utils.py", line 194, in _flatten_dense_tensors
    flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)
File "/usr/lib/python3.7/site-packages/torch/_utils.py", line 194, in <listcomp>
    flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)
AttributeError: 'NoneType' object has no attribute 'contiguous'

[schoennenbeck]

Here is an actual fix:

In apex/optimizers/fp16_optimizer.py line 147 currently reads

grads_groups_flat.append(_flatten_dense_tensors([p.grad for p in group]))

If you replace that with

grads_groups_flat.append(_flatten_dense_tensors([p.grad if p.grad is not None else p.new_zeros(p.size()) for p in group]))

you get rid of the bug. I would be happy to submit a pull request for this fix. However, I am not sure if this is really the optimal solution as we need to allocate a new all zeros tensor for each parameter that has ‘None’-gradient even though this clearly adds nothing to the computation. I haven’t found a way yet to get around that as all gradients and parameters get flattened so it is essential that we do have these zeros there.

[mcarilli]

@adihaviv Yes, there is a planned fix. I’m reworking FusedAdam so that it won’t require param flattening anymore (WIP branch is https://github.com/NVIDIA/apex/tree/multi_tensor_sgd) and None gradients should be acceptable in a single process.