why is trainer.current_epoch used for model checkpoint callback for Pytorch Lightning

1. In the model saving callbacks for Pytorch Lightning, why is step=trainer.global_step used instead of step=trainer.current_epoch? Using current_epoch seems more intuitive as the save checkpoints per trial will be named after the epoch of that trial and not the global ‘step’ of all trials.

[amogkam]

Ah I see, yes using that current_epoch as the dir name is fine as long as there is only one checkpoint per epoch. Your current setup looks good.

[turmeric-blend]

I have my callback like so: callback = TuneReportCheckpointCallback(..., on="validation_end"). And internally in the callback method I changed trainer.global_step to trainer.current_epoch for ease of readability of my files.

with tune.checkpoint_dir(step=trainer.current_epoch) as checkpoint_dir:   <--------
            trainer.save_checkpoint(
                os.path.join(checkpoint_dir, self._filename))

I was afraid I might break something (or worst have a saving bug without knowing it - eg maybe the latest best checkpoint won’t be saved if there already exists a checkpoint with the same name like you mentioned) since I am not so familiar with ray tune.

My guess was if my training/validation setup was in a way that it would never run into the case where multiple checkpoints would be made within a single epoch, then I wouldn’t have to worry about any checkpointing issues. Would having callback = TuneReportCheckpointCallback(..., on="validation_end") be enough to ensure this (so I can keep trainer.current_epoch)?