Bad nested CV example? It looks like each CV loop does the same exact split on the data

I think this example of nested cross validation (from the main site) isn’t implementing nested cross validation properly. The example uses the same exact data and does the same exact splits for each level of nested CV. This is inconsistent with my understanding of how nested CV should be implemented (see this question of mine in cross validated and the top answers).

# Loop for each trial
for i in range(NUM_TRIALS):

    # Choose cross-validation techniques for the inner and outer loops,
    # independently of the dataset.
    # E.g "LabelKFold", "LeaveOneOut", "LeaveOneLabelOut", etc.
    inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)
    outer_cv = KFold(n_splits=4, shuffle=True, random_state=i)

    # Non_nested parameter search and scoring
    clf = GridSearchCV(estimator=svr, param_grid=p_grid, cv=inner_cv)

    clf.fit(X_iris, y_iris) ## <<<-- LINE A
    non_nested_scores[i] = clf.best_score_

   # Nested CV with parameter optimization
    nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)  ## <<<-- LINE B
    nested_scores[i] = nested_score.mean()

note that the two lines above and that I called A and B set up the same K folds (i.e. same random state i) on the same exact data (X_iris and y_iris).

• clf.fit(X_iris, y_iris)
• nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)

This implementation isn’t consistent with a typical nested CV implementation, which would instead do a full loop of the inner CV in the data within each fold of the outer CV loop:

[amelio-vazquez-reina]

Phenomenal, thank you. This clarifies it:

cross_val_score fits the model again, for each split of the data.

I didn’t know cross_val_score would re-fit as opposed to just evaluate clf that was already fit, on the CV provided (sorry I always use GridSearchCV here so wasn’t sure about it). 👍

[jnothman]

I think we should just construct the grid search again in the cross_val_score call.