Unexpected order of tests using parameterized fixtures

Consider following code:

import pytest

@pytest.fixture(scope="function", params=(1, 2, 3))
def f1(request):
    pass
    
@pytest.fixture(scope="function", params=('a', 'b', 'c'))
def f2(request):
    pass

def test(f1, f2):
    pass

The order of the tests meets my expectations:

============================= test session starts =============================
platform win32 -- Python 2.7.12, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- c:\users\rbierbasz\.virtualenvs\core\scripts\python.exe
cachedir: .cache
rootdir: C:\Users\rbierbasz\repos\galaxy-core, inifile:
collected 9 items

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[3-a] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-c] PASSED

========================== 9 passed in 0.04 seconds ===========================

But when I changed fixtures’ scope from function to module:

import pytest

@pytest.fixture(scope="module", params=(1, 2, 3))
def f1(request):
    pass
    
@pytest.fixture(scope="module", params=('a', 'b', 'c'))
def f2(request):
    pass

def test(f1, f2):
    pass

it seems to get random:

============================= test session starts =============================
platform win32 -- Python 2.7.12, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- c:\users\rbierbasz\.virtualenvs\core\scripts\python.exe
cachedir: .cache
rootdir: C:\Users\rbierbasz\repos\galaxy-core, inifile:
collected 9 items

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[3-c] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-a] PASSED

========================== 9 passed in 0.04 seconds ===========================

The same happens for class and session scopes. It leads to fixtures being set up and torn down more times than necessary.

[rbierbasz]

@kchomski-reef thanks for the great insight. I haven’t checked reorder_items code yet, but your reasoning seems correct. I only think that the assumptions on which this functionality is based are naive - that all paremetrized fixtures have the same weight. In real world f1 fixture from my example can have much more expensive setup/teardown, or there can be more fixtures depending on it. It would be great if I could control the order somehow. I tried indirect parametrization:

import itertools
import pytest

f1_params = (1, 2, 3)
@pytest.fixture(scope="module", params=f1_params)
def f1(request):
    pass
    
f2_params = ('a', 'b', 'c')
@pytest.fixture(scope="module", params=f2_params)
def f2(request):
    pass

@pytest.mark.parametrize('f1,f2', itertools.product(f1_params, f2_params), indirect=True)
def test(f1, f2):
    pass

But it breaks fixtures’ scopes (#570). The only working solutions for me now is moving one parametrized fixture to higher scope:

import pytest

@pytest.fixture(scope="session", params=(1, 2, 3))
def f1_meta(request):
    return request.param

@pytest.fixture(scope="module")
def f1(f1_meta):
    pass
    
@pytest.fixture(scope="module", params=('a', 'b', 'c'))
def f2(request):
    pass

def test(f1, f2):
    pass

But it seems hacky and it’s not scalable. Is there any way of achieving it? Some hook maybe?

[gavincyi]

Good point to have a new issue rather than stuck in a closed ticket. #3393 created.