metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations

Both, python 2.7.13 and 3.5.3, pytest 3.6.1, no additional pytest plugins.

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}
def pytest_generate_tests(metafunc):
    if 'tested_branch' in metafunc.fixturenames:
        branches = KNOWN_BRANCHES.keys()
        metafunc.parametrize("tested_branch", branches, scope="session")

@pytest.fixture(scope="session")
def repo_preparation(tested_branch):
    """ If it would have a "session" scopoe then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[tested_branch] += 1
    return KNOWN_BRANCHES[tested_branch]

And the simplest reproduction:

# test_parametrization.py
import pytest

def test_nothing(repo_preparation):
    assert repo_preparation == 1  # that passes

@pytest.mark.parametrize("_", ["param_1", "param_2"])
def test_basic_parametrization(repo_preparation, _):
    # that causes parametrization confisuon
    assert repo_preparation == 1, "Repo prepared more than once."

Gives such an result:

$ pytest -v test_parametrization.py 
================================== test session starts ==================================
platform linux -- Python 3.5.3, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/kamichal/ecws/pyplayground, inifile:
collected 6 items                                                                       

test_parametrization.py::test_nothing[branch_1] PASSED                            [ 16%]
test_parametrization.py::test_basic_parametrization[branch_1-param_1] PASSED      [ 33%]
test_parametrization.py::test_nothing[branch_2] PASSED                            [ 50%]
test_parametrization.py::test_basic_parametrization[branch_1-param_2] PASSED      [ 66%]
test_parametrization.py::test_basic_parametrization[branch_2-param_1] FAILED      [ 83%]
test_parametrization.py::test_basic_parametrization[branch_2-param_2] FAILED      [100%]

======================================= FAILURES ========================================
_____________________ test_basic_parametrization[branch_2-param_1] ______________________

repo_preparation = 2, _ = 'param_1'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 2 == 1

test_parametrization.py:12: AssertionError
_____________________ test_basic_parametrization[branch_2-param_2] ______________________

repo_preparation = 3, _ = 'param_2'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 3 == 1

test_parametrization.py:12: AssertionError
========================== 2 failed, 4 passed in 0.03 seconds ===========================

Setting indirect=True in the metafunc.parametrize call fixes that problem, but there is no information that it’s required for larger scopes to work properly. There is only an advice to do so, but…

[winklerrr]

I came across the same bug, I think… When using metafunc.parametrize(scope=session) in combination with parametrized tests, all of my session scoped fixtures teared down after each test.

This behavior can be demonstrated very vividly with --setup-show:

SETUP    S session_fixture1 # <-- This fixture is generated with pytest_generate_tests
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test1()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2 
TEARDOWN S session_fixture1 # <-- Session scoped fixture gets tear downed
SETUP    S session_fixture1 
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test2()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2
TEARDOWN S session_fixture1

Workaround

@kamichal For me, the workaround with indirect=True also works in combination with parameters received from metafunc.config.getoption.

Just define a real fixture with the same name (in the example below my_fixture) and use the request fixture to return the indirect parametrized value:

def pytest_addoption(parser):
    parser.addoption("--my_parser_option")
    
def pytest_generate_tests(metafunc):
    if "my_fixture" in metafunc.fixturenames:
        metafunc.parametrize("my_fixture", metafunc.config.getoption("my_parser_option"), scope="session", indirect=True)

@pytest.fixture(scope="session")
def my_fixture(request):
    return request.param

def test_fixture(my_fixture):
    assert my_fixture == "world"

Then you can call it with pytest --my_parser_option hello, which will result in "hello" != "world" I hope this helps someone.

Regards Winklerrr

[The-Compiler]

I’m sorry to be nagging, but is this something that is going to be addressed someday?

As usual with open source projects, it will be addressed once someone feels like working on it (or someone pays someone to work on it).

Is this confirmed as a bug?

@RonnyPfannschmidt (a pytest core maintainer) said “that looks like a bug” above.