Checkpoint usability issues

There are a number of usability issues that have surfaced while documenting checkpoints.

• CheckpointLoader, CheckpointSaver and RunDirectoryUploader all have pretty out of date docstrings (Pain level: moderate)
• save_folder that is a relative path should be relative to CWD, not the run directory. (Pain level: high)
• Saving to object store requires the RunDirectoryUploader callback to be provided as well. This is not obvious and probably shouldn’t be required (Pain level: mild)
• Checkpoints should be saved as pytorch checkpoint files, not tarballs (Pain level: high)

Additional notes: Save folder Since the user is specifying the save_folder, it really should be respected. If they used a relative path, assume they meant relative to the CWD, not relative to some run directory. If we are concerned about checkpoint files overwriting others, we can either error or use the timestamp to ensure the filename is unique.

Saving to object stores This is not particularly high priority, but is a usability issue. To use the object store, we should do as much as possible to just specify the URI as the save_folder and not require a separate callback.

[hanlint]

I’ll add another one in (cc: @ajaysaini725 )

• For testing, we want to programmatically retrieve the paths of the saved checkpoints after training. e.g.:

trainer.fit()

saved_checkpoints = trainer.checkpoint_saver.saved_files  # absolute path

[ravi-mosaicml]

#421 and the new checkpoint API (https://www.notion.so/Checkpoint-API-Redesign-f0ee88d4bd5149b6a091589ed3f465f5) will fix the other issues identified in this issue. However, these changes are scheduled for v0.5, not v0.4.1