[s2s] distributed eval gets stuck on error w/ multigpu

examples/seq2seq/distillation.py and probably others remain hanging on internal error when run w/ multiple gpus (2 here):

rm -r /tmp/tmpqajqhzwo; PYTHONPATH="src" python examples/seq2seq/distillation.py --supervise_forward --normalize_hidden --label_smoothing=0.0 --eval_beams=1 --val_metric=loss --save_top_k=1 --adafactor --early_stopping_patience=-1 --logger_name=default --length_penalty=0.5 --cache_dir= --task=summarization --num_workers=2 --alpha_hid=0 --freeze_embeds --sortish_sampler --student_decoder_layers=1 --val_check_interval=0.5 --output_dir=/tmp/tmpqajqhzwo --no_teacher --fp16_opt_level=O1 --gpus=2 --max_grad_norm=1.0 --do_train --do_predict --accumulate_grad_batches=1 --seed=42 --model_name_or_path=sshleifer/tinier_bart --config_name= --tokenizer_name=facebook/bart-large --learning_rate=0.3 --lr_scheduler=linear --weight_decay=0.0 --adam_epsilon=1e-08 --warmup_steps=0 --max_epochs=2 --train_batch_size=1 --eval_batch_size=2 --max_source_length=12 --max_target_length=12 --val_max_target_length=12 --test_max_target_length=12 --n_train=-1 --n_val=-1 --n_test=-1 --student_encoder_layers=1 --freeze_encoder --data_dir=examples/seq2seq/test_data/wmt_en_ro --alpha_mlm=0.2 --alpha_ce=0.8 --teacher=sshleifer/bart-tiny-random

last output:

initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Traceback (most recent call last):
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 281, in <module>
    distill_main(args)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 269, in distill_main
    check_output_dir(args, expected_items=3)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/utils.py", line 641, in check_output_dir
    raise ValueError(
ValueError: Output directory (/tmp/tmpqajqhzwo) already exists and has 7 items in it (expected 3 items). Use --overwrite_output_dir to overcome.

and now it hangs, holding onto the gpu. Can’t even Ctrl-C the process - needed to suspend+kill manually.

I know that adding --overwrite_output_dir will remove the error, but this is not the issue. It shouldn’t hang on error (e.g. the test suite needs to continue running in such event).

@sshleifer

[sshleifer]

@williamFalcon @SeanNaren (lightning friends)

Do you guys have a clever way to collect failures in your multigpu tests? When something breaks, our multigpu test hangs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.