Improve tests using pytest parametrize

Improve tests using pytest parametrize

The idea is to improve our testing code using pytest.mark.parametrize.

Currently some tests are implemented as :

def test_something():

    def _test(a):
       # internal function doing some tests
       # For example, a dummy check below
       assert 0 < a < 5

    a = 1
    _test(a)

    a = 2
    _test(a, b, c)

We would like to implement that using pytest.mark.parametrize:

@pytest.mark.parametrize("a", [1, 2])
def test_something(a):
    # internal function doing some tests
    # For example, a dummy check below
    assert 0 < a < 5

Another example PR doing that for a test case: https://github.com/pytorch/ignite/pull/2521/files

What to do:

• Read CONTRIBUTING to see how to develop and execute a specific test: https://github.com/pytorch/ignite/blob/master/CONTRIBUTING.md#run-tests
• Explore test codebase and identify tests to improve. Please pay attention that some tests most probably can’t be reimplemented with pytest.mark.parametrize as they can generated random data. Let’s skip all _test_distrib_* tests as they can be particularly complicated to port.
• Once a test to improve is identified, comment out here about it and propose a draft code snippet of how you think to reimplement it using pytest.mark.parametrize.
• In parallel, you can try to reimplement it locally and test if the new test is still passing. Please make sure not to modify test assertions to make test pass as your PR wont be accepted.
• Send a draft PR

Thanks!

[vfdev-5]

Looks good to me @Ishan-Kumar2 , please send a PR 😃

[Ishan-Kumar2]

Hi @vfdev-5 Great idea of using pytest.mark.parameterize

A simple fix in https://github.com/pytorch/ignite/blob/3a286b1d13a4a0476b3ee8e8046f16465818c9f6/tests/ignite/handlers/test_time_limit.py#L18-L39 would be to write it as

@pytest.mark.parametrize("n_iters, limit",[(20, 10),(5,10)])
def test_terminate_on_time_limit(n_iters, limit):
        started = time.time()
        trainer = Engine(_train_func)

        @trainer.on(Events.TERMINATE)
        def _():
            trainer.state.is_terminated = True

        trainer.add_event_handler(Events.ITERATION_COMPLETED, TimeLimit(limit))
        trainer.state.is_terminated = False

        trainer.run(range(n_iters))
        elapsed = round(time.time() - started)
        assert elapsed <= limit + 1
        assert trainer.state.is_terminated == (n_iters > limit)

and move the _train function outside. Let me know what you think. Thanks!