Add optimal model size and stopping time feature

🚀 Feature request

The calculator blog post presented an automated way to find scaling laws with model size and compute budget on language modeling tasks. Adding it to the library would help save on training costs by picking an optimal model size and training time.

Motivation

Estimating how big of a model to use and how long to train for is more of an art than a science. An automated tool to perform that task would allow researchers and practitioners to concentrate on the the high-level parts of their projects as opposed to parameter tweaking.

Your contribution

I can submit a PR with my existing work, probably integrating it within Trainer and/or knocknock.

[TevenLeScao]

Great stuff, thank you! The energy estimates look 1000 worse than reality though, V100 running for 12 h should not consume 5432 kWh I think, else we’d be all dead. 5.4 kWh looks more reasonable.

Ah yes - I remembered having a doubt on that, I checked again the library we used to estimate those and there might have been a unit conversion error, I’ll fix that ASAP tomorrow!

Edit: it’s fixed, thank you @lopuhin !

[lopuhin]

Great stuff, thank you! The energy estimates look 1000 worse than reality though, V100 running for 12 h should not consume 5432 kWh I think, else we’d be all dead. 5.4 kWh looks more reasonable.