Should have option to use output weights when validating

We discussed this in passing before, but I think we need to discuss it directly. There are at least some applications where you really do need to use sample_weight in validation as well as in training. This should at least be an option (I’ll be building this for my uses either way, but would like to do so in such a way that I can merge it in to the master branch)

One use of sample_weight is to mask output values when you’re doing sequence to sequence learning with different length sequences. If you don’t also use sample_weight in validation, your loss will be hugely increased by all the meaningless masked output values.

See, for instance, the discussion happening over in #451

This applies to class_weight as well when you are classifying inputs and have very different numbers of training examples for each class, but want to remove that bias in your training. If I have 99 examples in class A for each example in class B I’d like to be able to, rather than throwing away 98% of my training examples to normalize class counts, rather pass in a class_weight of 1/99 to class A, and get equivalent validation loss.

[skarakulak]

It would be nice to have a class weighted validation loss without the need to modify the loss function itself.

[dc-ai]

It looks like class_weight still does not get factored into validation loss calculations?

I only found an option for adding in sample_weight to the fit command.