Allow disassembled use of check_estimator

Description

For my downstream project, I’m testing my classifier with check_estimator, and would like to see which of its individual tests failed. Doing so under nose works by using for check in _yield_all_checks(name, estimator): yield check, name, my_estimator (and a separate call to the class-level checks), but py.test won’t support yield-tests, and using _yield_all_checks to parameterize a test doesn’t work because the function requires the estimator instance.

Therefore the request is to provide all checks from check_estimator as a (non-private) iterable, so they can be used separately.

I brought this up in #10728 first, errorneously.

Versions

Linux-4.17.5-1-ARCH-x86_64-with-arch-Arch-Linux Python 3.6.6 (default, Jun 27 2018, 13:11:40) [GCC 8.1.1 20180531] NumPy 1.14.5 SciPy 1.1.0 Scikit-Learn 0.19.1

[rth]

I thought I had commented on this. I’m not sure I’m convinced it’s good to enable running individual tests because that will encourage not running all tests.

Thanks for the feedback. Still suppose one has a a single check that doesn’t pass in check_estimator in a sklearn-contrib project. With an iterative setup, one could just skip the check in question and mark it as TODO later, and run the rest of checks. Without it, in the current situation, one would just not use check_estimator altogether because it would fail. I’m sure why the latter situation would be better.

It would give developers in scikit-learn contrib projects more flexibility if needed, without waiting for our 6month-12month release cycle. Flexibility is one of the things that was mentioned as limiting factor in https://github.com/scikit-learn/scikit-learn/issues/6715. Of course in the ideal world check_estimator would take into account all the needs of contrib projects, but currently we are not there yet.

Also honestly it’s annoying to have one tests that runs several dozens checks. That’s why we are parametrizing them in scikit-learn (before with yields now with pytest). Giving contrib projects a possibility to do the same, and trusting developers not to misuse it, would be nice IMO.

[amueller]

Never mind, I was overthinking it, let’s go with https://github.com/scikit-learn/scikit-learn/issues/11622#issuecomment-405960168