[Train] Support custom checkpoint file names

Currently, the file names for Ray Train checkpoints are not customizable. They will always be of the format checkpoint_XXX.

Provide a way for the user to specify the name of the checkpoint file that they save.

One possible API is to allow the user to specify the checkpoint file name in train.save_checkpoint()

[matthewdeng]

Thanks @andrijazz for providing this context, I apologize if my comment came off as trying to restrict users!

The reason I asked was because with the current API, even with the ability to customize individual checkpoint names there may be some confusion since they will be written to the <logdir>/run_<run_id>/checkpoints directory, which may change over runs. Perhaps we need to allow customization of this directory name as well…

[andrijazz]

Being able to specify custom path and name of the checkpoints would be great.

One other use-case that comes to mind is that user might want to store checkpoints outside of ray generated folders … for example wandb creates its own dirs and automatically uploads all files stored in those dirs to the cloud after run is finished. User might want to store checkpoints inside wandb dir because he can easliy browse through them on the wandb web app and decide which one to use based on the plots.