[s2s] Trainer vs PTL timings

For the following two commands,

• PTL finishes: 2.01 it/s, ~3H, 21.32 Rouge
• Trainer: 1.0 it/s, roughly 5.5H, 21.36 Rouge

I wanted to report this so I don’t lose track of it. Looked at the code, and don’t see any obvious issue, besides that the slowdown is suspiciously close to 2x.

Any idea @patil-suraj ?

PTL Command

export BS=32
export GAS=1

python finetune.py \
    --learning_rate=3e-5 \
    --fp16 \
    --gpus 1 \
    --do_train \
    --do_predict \
    --val_check_interval 0.25 \
    --n_val 500 \
    --num_train_epochs 2 \
    --freeze_encoder --freeze_embeds --data_dir cnn_dm \
    --max_target_length 142 --val_max_target_length=142 \
    --train_batch_size=$BS --eval_batch_size=$BS --gradient_accumulation_steps=$GAS \
    --model_name_or_path sshleifer/student_cnn_12_6 \
    --tokenizer_name facebook/bart-large \
    --warmup_steps 500 \
    --output_dir distilbart-cnn-12-6

Trainer command

same as builtin_trainer/train_distilbart_cnn.sh:

export BS=32
export GAS=1
export m=sshleifer/student_cnn_12_6
export tok=facebook/bart-large
export MAX_TGT_LEN=142

python finetune_trainer.py \
    --model_name_or_path $m --tokenizer_name $tok \
    --data_dir cnn_dm \
    --output_dir distilbart-cnn-12-6-trainer --overwrite_output_dir \
    --learning_rate=3e-5 --sortish-sampler \
    --warmup_steps 500 \
    --fp16 \
    --n_val 500 \
    --gradient_accumulation_steps=$GAS \
    --per_device_train_batch_size=$BS --per_device_eval_batch_size=$BS \
    --freeze_encoder --freeze_embeds \
    --num_train_epochs=2 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 142 --val_max_target_length $MAX_TGT_LEN --test_max_target_length $MAX_TGT_LEN \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --sortish_sampler

[patil-suraj]

I’m also experiencing slow down on TPU’s, didn’t run the new changes on GPU yet. I"ll investigate this

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.