Clarification questions

Hi, I have a few questions regarding TransCoder’s training data and optimization setting.

1. From the paper, it is clear that TransCoder is trained using Standalone functions during the DAE+BT training stage. But is TransCoder only trained using Standalone functions in the MLM stage too?
2. During the MLM stage, only the encoder part of TransCoder is pre-trained, right?
3. For the MLM pre-training, max_epoch and epoch_size are set to 100k. If I understand correctly, epoch_size basically refers to the number of instances used in each epoch. Is it correct? Also, for MLM pre-training, the following are set:

--validation_metrics _valid_mlm_ppl \
--stopping_criterion '_valid_mlm_ppl,10' 

So, I am assuming TransCoder pre-training is stopped based on the stopping_criterion. Before, the MLM pre-training was stopped, how many optimization steps were executed?

1. Unlike the MLM pre-training stage, for the DAE+BT stage training, there is no stopping_criterion is set. And the epoch_size was set to 50000 and the max_epoch was set to 10000000. So, when the training stops? How many optimization steps were executed during this stage?

[brozi]

It’s per GPU so yes the actual batch size is 32 * 32

[brozi]

The epoch size is supposed to be the number of samples you train on. However there is something that’s a bit confusing in our code. We always use increase the sentences counter by the batch_size parameter when training but it’s not always the actual batch size.

• for stream datasets (e.g. for MLM) we use the batch parameter so epoch_size is really the number of samples (per GPU so you need to multiply by 32)
• for DAE and BT we fix the number of tokens per batch instead of the epoch size to avoid OOMs due to batches of varying sizes so the batch size is not constant. In that case epoch_size is num_updates/32 (32 is the default batch size parameter), which doesn’t really cause any issues but is quite confusing.

I’ll think about changing the behaviour of this parameter to have something more coherent in a way that minimizes the changes people will need to make to the parameter.