Add `return_predictions` option to the model_selection.cross_validate() API

The cross_validate(), cross_val_predict(), and cross_val_score() methods from the model_selection module provide highly compact and convenient API for most daily works. Yet often times, one would like to get 1) multiple scores for both train and test, and 2) predictions for all samples using cross validation, which is an aggregation of test predictions from all folds.

Current API doesn’t seem to allow both goals to be achieved in a single cv run. Is it possible to add something like a return_predictions option to the model_selection.cross_validate() API, as in here, to simply allow access to the results (which I believe have already been computed inside the function)?

Thank you!

[amueller]

I don’t really like returning the indices and models. You’re still requiring the user to reimplement quite a bit, why wouldn’t they just completely reimplement it as @mostafahadian did?

I would favor adding a return_predictions (and possibly predict_method) to cross-validate, which would provide per-fold predictions. I think the user can be trusted to hstack them.

[after reading the above more carefully: as usual I agree with @jnothman]

[jnothman]

I think wanting to get predictions as well as scores is a common use-case, albeit open to misuse. To some extent I’d rather have return_predictions which would return test set predictions as well as indices (under a separate key) rather than return_includes. One reason is that there’s no harm including CV indices if returning predictions, and it might help guide the users towards reasonable usage patterns. One reason is that some things should be returned without being asked for, and multi-metric scoring naturally expands the set of returned keys. One reason is that it’s better for return_train_score to be available consistent with *SearchCV.