fixture assert is getting overwritten in pytest_collection_modifyitems

Please create 2 files in any folder with below codes:

1. test_py_issue.py

import pytest

def idfn(fixture_value):
    return 0

def create_test():
    import somerandomPACKAGE # inserting error in fixture
    return ["a", "b", "c"]

@pytest.fixture(scope="function", params=create_test(), ids=idfn)
def op_test_config(request):
    return request.param

def test_py_collect_issue(op_test_config):
    print(f"\nHello I am a debug test! and config: {op_test_config}")

1. conftest.py

def pytest_collection_modifyitems(config, items):
    if len(items) !=3:
        print(len(items))
        assert 0, "expected 3 tests to be created!" # commenting this line will give actual error

Here the output is

INTERNALERROR>     assert 0, "expected 3 tests to be created!" # commenting this line will give actual error
INTERNALERROR> AssertionError: expected 3 tests to be created!
INTERNALERROR> assert 0`
But the issue raised in fixture , which is import issue. so pytest should print first raised issue. 
`Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_py_issue.py:11: in <module>
    ???
test_py_issue.py:7: in create_test
    import somerandomPACKAGE # inserting error in fixture
E   ModuleNotFoundError: No module named 'somerandomPACKAGE'`

pip list:

Pillow                  9.2.0              
pip                     20.0.2             
platformdirs            2.5.2              
pluggy                  0.13.1             
psutil                  5.4.2              
py                      1.11.0                         
pyasn1                  0.4.8              
pycodestyle             2.9.1              
pyflakes                2.5.0              
pylint                  1.9.3              
pymongo                 3.7.2              
pyparsing               3.0.9              
pytest                  6.2.2              
pytest-forked           1.3.0              
pytest-html             3.1.1              
pytest-logger           0.5.1              
pytest-metadata         2.0.2              
pytest-random-order     1.0.4              
pytest-repeat           0.9.1              
pytest-timeout          2.0.2              
pytest-xdist            2.4.0              
python-dateutil         2.8.2

Downloads.zip

[RonnyPfannschmidt]

The error is stored as a report to be printed in the summary, your assertion in the place where you should not asset breaks pytest before it gets to the printing

[JosiasAurel]

Hey, I just tested your code and it works for me. Here is what I have In test_py_issue.py

import pytest

def idfn(fixture_value):
    return 0

def create_test():
    import numpy as np # inserting error in fixture
    return ["a", "b", "c"]

@pytest.fixture(scope="function", params=create_test(), ids=idfn)
def op_test_config(request):
    return request.param

def test_py_collect_issue(op_test_config):
    print(f"\nHello I am a debug test! and config: {op_test_config}")

In conftest.py

def pytest_collection_modifyitems(config, items): 
    if len(items) !=3: 
        print(len(items)) 
        assert 0, "expected 3 tests to be created!" # commenting this line will give actual error