Integration testing for downstream dependencies

We need a new Continuous Integration configuration to launch the tests of some popular downstream projects.

Candidate projects:

• xgboost and or lightgbm as very popular third party libraries
• skorch
• imbalanced-learn not necessarily as popular as others but with high use of sklearn’s API
• autosklearn?
• any other suggestion?

Those test would not be run in PRs unless we use a specific tag in the commit message for instance. This test setup would be run by maintainers in the release branches.

We do something similar in the cloudpickle project with the [ci downstream] commit tag:

https://github.com/cloudpipe/cloudpickle/blob/master/.travis.yml#L37-L70

For scikit-learn this would probably need a bit more scripting as the build is more complex than for a pure python project such as cloudpickle but the general idea would be the same.

[jnothman]

Frankly the downstream packages should be doing this integration testing by testing various use cases and checking with scikit-learn master. We should be encouraging downstream packages to include such integration tests (e.g. various usages of their estimators with grid search).

But it’s reasonable that we at least run a suite of integration tests as part of a release cycle. Putting this in Azure makes it easier for our maintainers.

I don’t mind if we also run a Cron job but then we will have to deal with more false alarms (e.g. downstream test suites failing due to poor or delayed management of their other dependencies).

[glemaitre]

I think that we can address the following when we have a more granular check_estimator to be sure that we are testing the right things for these third-party libraries. xref: https://github.com/scikit-learn/scikit-learn/issues/16241