[Feature] Hook to implement early stopping

Short Question Description

I would like to implement a hook for the user to be able to implement his own stopping strategy. Is this interesting for you? How would I go about implementing the hook myself?

Context Information

First off, I really like the project and I’m very much impressed with what you have accomplished. The autoML engine works extremely well for our use cases.

Different datasets require different training lengths. In some cases, I noticed that the autoML engine already finds the most optimal configuration in a matter of seconds, whereas for others, longer training times do benefit the performance. Without knowing the dataset in advance, it is hard to find the optimal training time - we are trying to minimize computation time while keeping the same model performance.

This could be done by providing a hook to the user that is called after every new model is trained. I would then check the new model’s performance with the best one so far, and make a heuristic decision to continue training based on this. The time_left_for_this_task would still be the maximum training time - the hook would thus implement some sort of early stopping strategy.

Similar Work

I did not find a similar example/tutorial in the documentation, nor a similar GitHub Issue.

[mfeurer]

While functionally it’s no different from the current parameter to pass callbacks, it’s a lot clearer as an entry point to using this functionality.

Wouldn’t it maybe suffice to rename the current argument and improve its documentation?

[eddiebergman]

Raising this in a new issue so we can clearly state what needs to be done on our end. Please see #1304