Resuming Training

My model and optimizer are prepared using .prepare(), my scheduler is left without “preparation” as was done in the nlp_example.py

If I want to resume training, do I need to use the .prepare() function again after loading the previously saved model?

Or would I need to use the .prepare() function only if I loaded the trained model before calling .prepare()?

[Yaicky]

Hi, @sgugger . I find one accelerator.gather() issue related to saving method in distributed evaluation. if I accelerator.save(wrapped_model), and for loading, prepare, load(wrapped_model_ckpt), just like you said before on this issue page, when I appended accelerator.gather()(same codes in cv_example.py) to my eval function, I got stuck at the second iteration, no errors, no exits, just stuck(I use tqdm, progressbar didn’t go.)

So I followed your official document to save unwrapped_model first, thing goes well. Maybe it’s not an issue cause saving unwrapped_model is recommended, just FYI. 😃

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.