SLEP006 - Metadata Routing task list

This issue is to track the work we need to do before we can merge sample-props branch into main:

• Based on this prototype: https://github.com/scikit-learn/scikit-learn/pull/16079
• Merge https://github.com/scikit-learn/scikit-learn/pull/22083 into sample-props. This PR only touches BaseEstimator and hence consumers. It does NOT touch meta-estimators, scorers or cv splitters.
• Work on splitters and merge that into sample-props: https://github.com/scikit-learn/scikit-learn/pull/22765
• Add new terms to the glossary: router, consumer, metadata
• Work on scorers and merge that into sample-props ~(note that this involves an ongoing discussion on whether we’d like to mutate a scorer or not)~: https://github.com/scikit-learn/scikit-learn/pull/22757
• Get backward compatibility mechanisms in place for meta-estimators which already route given metadata: https://github.com/scikit-learn/scikit-learn/pull/22986
• Work on meta-estimators: this involves writing get_metadata_routing in easy cases, and a whole lot more in cases where we’d like to keep backward compatibility in parsing input args suchs as estimator__param in Pipeline
• Once all the above is merged into sample-props, do a few tests with third party estimators to see if they’d work out of the box. Note that consumer estimators should work out of the box as long as they inherit from BaseEstimator and their fit accepts metadata as explicit arguments rather than **kwargs.
• Check whether a library such as cuML could vendor _metadata_requests.py and work with scikit-learn meta-estimators w/o depending on the library.
• Refactor tests: https://github.com/scikit-learn/scikit-learn/issues/23918
• Merge sample-props into main: https://github.com/scikit-learn/scikit-learn/pull/24027

Enhancements:

• https://github.com/scikit-learn/scikit-learn/pull/24023
• https://github.com/scikit-learn/scikit-learn/issues/18936

Open issues:

• https://github.com/scikit-learn/scikit-learn/issues/22987: https://github.com/scikit-learn/scikit-learn/pull/24585
• https://github.com/scikit-learn/scikit-learn/issues/22988: https://github.com/scikit-learn/scikit-learn/pull/23342
• https://github.com/scikit-learn/scikit-learn/issues/23920
• https://github.com/scikit-learn/scikit-learn/issues/23928
• https://github.com/scikit-learn/scikit-learn/issues/23933

Our plan is to hopefully have this feature in 1.1, which we should be releasing in late April/early May.

Here’s a list of meta-estimators which need to be updated:

• AdaBoostClassifier: https://github.com/scikit-learn/scikit-learn/pull/24026
• AdaBoostRegressor: https://github.com/scikit-learn/scikit-learn/pull/24026
• BaggingClassifier: https://github.com/scikit-learn/scikit-learn/pull/24250
• BaggingRegressor: https://github.com/scikit-learn/scikit-learn/pull/24250
• CalibratedClassifierCV: https://github.com/scikit-learn/scikit-learn/pull/24126
• ClassifierChain: #24443
• ColumnTransformer
• ElasticNetCV
• FeatureUnion
• GraphicalLassoCV
• GridSearchCV
• HalvingGridSearchCV
• HalvingRandomSearchCV
• IterativeImputer
• LarsCV
• LassoCV
• LassoLarsCV
• LogisticRegressionCV #24498
• MultiOutputClassifier https://github.com/scikit-learn/scikit-learn/pull/22986
• MultiOutputRegressor https://github.com/scikit-learn/scikit-learn/pull/22986
• MultiTaskElasticNetCV
• MultiTaskLassoCV
• OneVsOneClassifier
• OneVsRestClassifier
• OrthogonalMatchingPursuitCV
• OutputCodeClassifier
• Pipeline: https://github.com/scikit-learn/scikit-learn/pull/24270
• RANSACRegressor
• RFE
• RFECV
• RandomizedSearchCV
• RegressorChain: #24443
• RidgeCV
• RidgeClassifierCV
• SelectFromModel
• SelfTrainingClassifier
• SequentialFeatureSelector
• StackingClassifier
• StackingRegressor
• TransformedTargetRegressor
• VotingClassifier
• VotingRegressor
• cross_validate
• cross_val_score
• cross_val_predict
• permutation_test_score
• learning_curve
• validation_curve

cc @jnothman @thomasjpfan @lorentzenchr

[adrinjalali]

I spent a few days trying to write common tests for meta-estimators but that didn’t go anywhere. After being stuck for a while, @thomasjpfan and I spent some time last week together and we decided to start with simple individual tests, starting for one meta-estimator, and then refactor the tests later when we find recurring patterns.

Right now I’m working on multioutput meta-estimators and should have a PR coming today.

[adrinjalali]

Also updated the list, it should be quite complete now I think.