[Feature] Terminate on NaN handler

What do you think about to add a handler that stops the training when Nan is encountered in the loss output? It is not that complicated to check with a custom handler but as such handlers can be found in other popular frameworks, maybe it worse to add something similar in ignite ?

[vfdev-5]

A solution can be proposed using anomaly detection introduced in pytorch 0.4.1

[vfdev-5]

As discussed in slack, probably a correct behaviour in a situation with NaN/Inf loss is too stop the training:

NaN+anything=NaN, so you’d just continue getting NaN…for it to be useful you need a mechanism to roll-back to a previous value which was not NaN