80min training time to fine-tune BERT-base on the SQuAD dataset instead of 24min?

I just fine-tuned BERT-base on the SQuAD dataset with an AWS EC2 p3.2xlarge Deep Learning AMI with a single Tesla V100 16GB:

I used the config in your README:

export SQUAD_DIR=/path/to/SQUAD

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

It took 80min. According to your README:

This example code fine-tunes BERT on the SQuAD dataset. It runs in 24 min (with BERT-base) or 68 min (with BERT-large) on a single tesla V100 16GB.

How to explain this difference? Is there any way to accelerate the training to 24min as well? Thanks

[thomwolf]

You should use 16bit training (--fp16 argument). You can use the dynamic loss scaling or tune the loss scale yourself if the results are not the best.

[thomwolf]

You can have a look at the readme examples but it should be a lot higher, around 88-90. Maybe your batch size is too small, look at the readme for more information.