Possible Feature Request(?): AUC metric and optimism via bootstrapping

Describe the workflow you want to enable

In the computation of AUC, with Confidence intervals, one may either use cross-validation hold-out data (e.g. any of the <X>-Fold functions in sklearn).

However, this procedure can be a bit problematic especially in low-sample data settings. In these low-sample data settings, it is sometimes suggested to instead use bootstrapping to estimate the “optimism” of the model, which can then be corrected for in the roc_auc_score.

Describe your proposed solution

Under the inspection, or utils module perhaps, a compute_metric_optimism() function could be used perhaps? This can be extended to metrics beyond just AUC.

Some pseudo-code:

def compute_metric_optimism(estimator: BaseEstimator, n_bootstrap: int, scoring: <scoring_func>, n_jobs=None):
    # compute the original metric on original data  
    orig_score = scoring()

    for boot_idx in range(n_bootstrap):
        # fit model
        estimator.fit()

        # compute score  on sample
        score = scoring()

    # compute optimism
    optimism = <average over the bootstrap samples>

    # compute optimism adjusted metric
    return orig_score - optimism

Describe alternatives you’ve considered, if relevant

If maintainers feel it is not necessary, then it is always possible to do it outside of sklearn, either w/ 3rd party packages, or self-implemented.

Additional context

For some reference on the procedure in R: https://thestatsgeek.com/2014/10/04/adjusting-for-optimismoverfitting-in-measures-of-predictive-ability-using-bootstrapping/

Happy to try a PR if this goes off.

[adrinjalali]

Considering cross_validate doesn’t require the splits to be disjoint sets, and that it returns the test scores as well as the train scores, a very short example in the example gallery would be welcome.

[adrinjalali]

Closing as WONTFIX, please refer to the comments on the PR