LongT5ForConditionalGeneration NAN losses with bf16

System Info

transformers version: 4.23.0.dev0 torch version: 1.12.1 OS: Ubuntu 20 Cuda: 11.6

The problem is that LongT5 is supposed to work with bf16=True, but it doesn’t. It is known that fp16 should fail in this, and I have tried it and effectively fails. However, Longt5 is supposed to be trained on bf16, therefore it would be expected that turning bf16 to True would work. My training arguments look like this:

{
        "evaluation_strategy": "epoch", 
        "num_train_epochs": 4,
        "do_train": True,
        "do_eval": False,
        "eval_steps": 2,
        "logging_strategy":"epoch",
        "save_strategy": "epoch",
        "save_total_limit": 4,
        "seed": 69,
        "bf16": True, 
        "dataloader_num_workers": 32,
        "adam_epsilon": 1e-8,
        "adam_beta1": 0.9,
        "adam_beta2": 0.999,
        "group_by_length": False,
        "gradient_checkpointing": False,
        "lr_scheduler_type": "linear",
        "learning_rate": 1e-4,
        "per_device_train_batch_size": 1,
        "per_device_eval_batch_size": 1,
        "gradient_accumulation_steps": 64,
        "warmup_ratio": 0.08
    }

Who can help?

@patrickvonplaten

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

If you need a script for reproduction please let me know.

Expected behavior

Longt5 (as I understand from the forum etc) should work with bf16.

[alexvaca0]

Great, with the last version it does work!! Thank you very much for helping me ! @ArthurZucker

[alexvaca0]

Okay thanks! Let me know if I can help in some way… 😃 @ArthurZucker