Add option to compute permutation_importance on a random subset of rows

For very large data sets, computing the feature importance on a random subset of rows instead of all rows can spead up the calculation dramatically. Since permutation importance runs several iterations per variable, it makes sense to draw a new subset each time to cover a larger part of the data.

The change would require sklearn.inspection.permutation_importance to have an additional argument specifying a maximum number of rows. This functionality is available in other packages, for example, in R: https://rdrr.io/cran/ingredients/man/feature_importance.html

Proposed solution

To be more precise I’d modify _calculate_permutation_scores like this:

def _calculate_permutation_scores(self, estimator, X, y, sample_weight, col_idx,
                                      random_state, n_repeats, scorer, max_rows):
        X_permuted = X.copy()
        y_mod = y.copy()
        n_rows = X.shape[0]
        if max_rows is not -1 & n_rows>max_rows:
            rng = np.random.default_rng()
            sample_rows = rng.choice(n_rows, max_rows)
            X_permuted = X_permuted.iloc[sample_rows]
            y_mod = y_mod.iloc[sample_rows]
        scores = np.zeros(n_repeats)
        shuffling_idx = np.arange(X_permuted.shape[0])
        for n_round in range(n_repeats):
            random_state.shuffle(shuffling_idx)
            if hasattr(X_permuted, "iloc"):
                col = X_permuted.iloc[shuffling_idx, col_idx]
                col.index = X_permuted.index
                X_permuted.iloc[:, col_idx] = col
            else:
                X_permuted[:, col_idx] = X_permuted[shuffling_idx, col_idx]
            feature_score = _weights_scorer(
                scorer, estimator, X_permuted, y_mod, sample_weight
            )
            scores[n_round] = feature_score

        return scores

I am happy to make a PR but wanted to have the opinion from the core developers first.

[o1iv3r]

Subsampling the data before sending it to the function means calculating all statistics on this subset only, likely introducing quite a bias. Sub sampling within an iteration is cleaner imho.

An alternative is calling the function with n_repeats equal to 1 several times, manually aggregating the results. But then the parallelisation by variable becomes quite inefficient.

[cmarmo]

If I’m not mistaken this issue can be close as solved by #20431. @knoam feel free to open a new one with your proposition. Thanks!