Add interrupt handlers

Since many experiments can take a while and will be running on an environment where the user does not have a lot of control (e.g., a shared cluster), it would be interesting to have interrupt handlers to do something in case the operating system sends a signal to kill the process during the execution of the fit method. Blocks does this by using the signal module (which is part of the standard library). That way, you can save the current model state (using pickle, for example) before letting the OS kill your process. Would you be interested in adding something similar to the fit method in model.py?

[halflings]

Any chance this can be reopened? It’s fair that programs not ending on SIGINT are annoying, but I think that somebody that would add this callback explicitly would be very much aware of this fact, and that they’ll just need to send SIGINT twice for the running command to stop immediately.

When iterating with a model running in the cloud, I need to run some clean-up code once training ends, and I often realize that I used too many epochs too late into this process. This would help with these cases.

[asafh]

I’m made a small keras callback that listens for signal (SIGINT by default, using the signal module) and stops the training once this epoch is complete. Would it make sense to open a pull request adding it to callbacks.py? I think it’s a decent solution to allow you to easily interrupt a long training session, and have your code afterwards execute (e.g. save model then exit).

Related, would it be save to call model.save mid-training?