[Feature request] Save a checkpoint when interrupting the training (ctrl - c)

🚀 Feature Description Hi, Sometimes while training, I need my GPU (eg to do some work with Whisper, or because I need to switch off the computer). So I have to interrupt the training and sometimes it is right between 2 checkpoints (eg checkpoints are saved every 10k iterations and it is 7k after the previous saved checkpoints). So in this case I would loose all the training that has been achieved after the previous checkpoint.

Consequently it would be more comfortable that a checkpoint is saved when I interrupt the training so that I can then restore the training right from this checkpoint.

Solution

When the training process is interrupted (ctrl-c) make coqui save a checkpoint at the current step (as it does when save_step is reached).

Alternative Solutions

I could lower save_step but then checkpoints are too near to each others.

Additional context

[p0p4k]

Yes, I am on discord in the coqui channel. I’ll ping there.

[erogol]

@p0p4k found you here… are you on discord? I want to add you to the contributor’s list if you want.