Large models aren't converge while fine-tuning

I tried to fine-tune XLM-Roberta Large model on Google Colab environment for 3 epochs using 1e-5 learning rate, 16 batch size, 2 accumulative steps and 120 warmup steps. But the loss didn’t converge and the model gives random predictions after the fine-tuning.

I used the sentence pair minimal example as a starting point.

Do you have any idea?

[antonyscerri]

in my case carefully adjusting the learning rate (with the existing scheduler) along with the number of epochs (as well as the previous increase in batch size) allowed me to get much better results (beating those with the base model to date). So there doesn’t appear to be anything fundamentally wrong with the pre-trained model or core model code. Seems you need to do a broader sweep of the parameters in your case (assuming no data issues etc).

[kinoute]

I understand that. But you should add checkpoints within epochs. This won’t improve your model but it will give you more insights on what’s going on really since you will get way more metrics at different moments of the process.

I’m about to re-run XLM Roberta on my binary classifier and I will report what worked for me.