Out of Fold score calculation for Cross validation

Describe the workflow you want to enable

What roughly happens in cross-validation as of now:

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section)

Describe your proposed solution

What OOF would do:

data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))

Describe alternatives you’ve considered, if relevant

No response

Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

[glemaitre]

I am -1 for adding another parameter to cross_validate

@thomasjpfan I might be interesting to be able to return the predictions in cross_validate. Indeed, it might be useful in plotting displays if we provide the cv_results in some cases. However, it should not be returned by default and should as well come with a big warning about when to use it.

[aadarshsingh191198]

@thomasjpfan @glemaitre