T5 fp16 forward yields nan

🐛 Bug

Information

Model I am using (Bert, XLNet …): T5

Language I am using the model on (English, Chinese …): English

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

I use pytorch-lightning to manage fp16. This is the minimal example that reproduces the result.

from transformers import T5Model, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-base")
model = T5Model.from_pretrained("t5-base").cuda().half()
text = "hello world!"
inputs = tokenizer.encode(text, return_tensors="pt").cuda()
out = model(input_ids=inputs, decoder_input_ids=inputs)
print(out[0][:, :, :10])

output:

tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]]], device='cuda:0',
       dtype=torch.float16, grad_fn=<SliceBackward>)

Expected behavior

Get non-nan values.

Environment info

• transformers version: 2.9.0
• Platform: Linux-4.15.0-88-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.6
• PyTorch version (GPU?): 1.4.0 (True)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

[patrickvonplaten]

Thanks for the detailed error description @binshengliu! I linked a PR that should fix it 😃

[SamsTheGreatest]

Same when fine-tuning GPT Neo.