Optional logging of validation loss (and other metrics) in KerasModel

KerasModel (and TensorGraph) currently don’t support periodic logging of validation loss. Would it be a good idea to have this in the fit_generator and fit API?

This would be an optional argument with the modified API looking something like:

def fit(self, dataset, nb_epoch=10, max_checkpoints_to_keep=5, 
      checkpoint_interval=1000, deterministic=False, restore=False, 
      submodel=None, val_dataset=None, eval_interval=1000, **kwargs):

It would make it easier to do things like early stopping, which was done in the ChemNet Transfer Learning paper, and in general can be useful as well

[peastman]

Agreed, this would be very useful. We should give some thought to how this should work. For fit(), it could work to just provide another dataset to use for validation. For fit_generator() that isn’t necessarily possible. After all, one of the purposes of that method is to support models that take multiple inputs and therefore require more than just the X array from a dataset.

You also mentioned the possibility of tracking other metrics, which would also be useful. So perhaps we can have a unified mechanism that supports all of those things.

[vsomnath]

I borrowed the BaseLogger term from Keras Callbacks, but it was motivated by your invoked “after every step” statement:

A callback can be any callable object. It gets invoked after each step, with the KerasModel being passed as the only argument.

Using the same idea as Keras, this CallBack would compute the loss at every step, and then display every certain iterations. This is used by default in every model. It can slso compute some metrics on the training set, if needed.

In case of Keras (https://github.com/keras-team/keras/blob/master/keras/callbacks.py), the EarlyStoppingCallback keeps track of the best weights if a toggle is turned on. So having one class should be sufficient.