Convert some `unittest` tests to `pytest` style

Some of our tests follow the unittest structure of creating tests. While these are fine, we want to migrate our tests to be in the more functional style of pytest.

For example, test_classification which tests the classification pipeline adopts the unittest style of class structured tests in which you must understand the whole setup to often understand one test.

In contrast, a more functional pytest looks like test_estimators.py::test_fit_performs_dataset_compression. Here we can define the test entirely through parameters, specify multiple parameter combinations, document which ones we expect to fail and give a reason for it. Because the test is completely defined through parameters, we can document each parameter and make the test almost completely-transparent by just looking at that one test.

We would appreciate any contributors on this side who want to get more familiar or are already familiar with testing. Some familiarity or willingness to learn pytest’s @parametrize and @fixture features is a must. Please see the contribution guide if you’d like to get started!

[eddiebergman]

Hi @shantam-8,

So I took at test_metrics and I have a few pointers.

• Thankfully the tests look very straightt forward, they create some dummy data, they create a scorer and then they make sure it gives the correct output.
• We want to do the following migration:

# From
import unittest

class TestScorer(unittest.TestCase):

    def one(self):
        ...

# To
def test_one() -> None:
    ...

• You can use the following replacements:

# From
self.assertAlmostEqual(a, b)
self.assertTrue(t)

# To
from pytest import approx
assert a == approx(b)
assert t is True

• Anywhere we use something like _ProbaScorer(...), we should instead use the public API way of making a Scorer, i.e. make_scorer(...). This to ensure the public API works as intended. https://github.com/automl/auto-sklearn/blob/ee0916ece79fd8f27059c37acf08b695e2c81ac8/autosklearn/metrics/__init__.py#L242-L253

• It would be nice to have a docstring below each test saying what it tests:

def test_sign_flip(...) -> None:
    """
    Expects
    -------
    * Flipping greater_is_better for r2_score should result in flipped signs of
    its output. 
    """

• If there’s any test that seems like it’s actually testing 2 things or many things at once, feel free to split them up into seperate tests, use your own judgement and we’ll just discuss it during the PR 😃

Please feel free to make any other changes that make sense in your judgement, as long as the core thing being tested is still being tested! You should also check out the CONTRIBUTING.md, especially the section on testing to understand how you can run only the tests you care about. Using pytest test/test_metric should be enough but there’s more info in the guide if you need.

You can continue asking things here or make an initial PR and we can discuss further there at any point!

Thanks for thinking to contribute 😃

Best, Eddie

[shantam-8]

Really thanks a lot for your help! I shall open a PR asap.