Note about appropriate and inappropriate uses of cross_val_predict

Users are tempted to run metric(y_true, cross_val_predict(...)). This is not equivalent to cross_val_score(...) which takes an average over cross-validation folds; nor is it a standard or appropriate measure of generalisation error.

I consider valid uses of cross_val_predict to be:

• visualisation of training set predictions
• model blending: you have an ensemble in which a “downstream” estimator learns from the predictions of an “upstream” estimator, so you should train the downstream not on the true labels, nor on the upstream estimator’s predictions on its training data as both are too biased. cross_val_predict is not biased in the same way.

Do you agree with this summary, @GaelVaroquaux?

[jnothman]

On the contrary, that distribution is a good indication that either your cv strategy is inappropriate or that you do not have enough data to get a good estimate of generalisation error or that your generalisation error is truly highly variable.

[reiinakano]

I agree that it doesn’t make sense for all cases, but why would it not make sense for something like accuracy?

If your CV method has imbalanced folds, for instance:

Acc: 0.67 on 20 test samples Acc: 0.7 on 20 test samples Acc: 0.8 on 10 test samples

simply averaging them would give the third fold disproportionate importance in the final accuracy value.

Wouldn’t running accuracy_score(y_true, cross_val_predict(...)) give a more accurate value in this case?