[v7.0.0 regression?] AssertionError: previous item was not torn down properly

I’m trying to update from pytest v6.2.5 to v7.0.0 via https://github.com/cherrypy/cheroot/pull/475 and it fails the testing weirdly with $sbj.

The assertion error in question comes from the pytest’s internals. Looking at the traceback promptly suggests that it’s happening when one test module is being set up but the previous module hasn’t been torn down completely: https://github.com/pytest-dev/pytest/blob/f28421cc7068b13ba63c1f60cc21f898cccea36c/src/_pytest/runner.py#L489.

I don’t know how it’s possible so any debugging pointers are welcome. Also, I don’t see this problem locally (although, I have to admit that I haven’t checked all of the possible runtime combinations).

Here’s what I managed to extract so far:

==================================== ERRORS ====================================
_________________ ERROR at setup of test_ssl_adapters[builtin] _________________
[gw1] linux -- Python 3.10.2 /home/runner/work/cheroot/cheroot/.tox/python/bin/python

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f06884add80>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

cls        = <class '_pytest.runner.CallInfo'>
duration   = 0.0007801090000043587
excinfo    = <ExceptionInfo AssertionError('previous item was not torn down properly') tblen=6>
func       = <function call_runtest_hook.<locals>.<lambda> at 0x7f06884add80>
precise_start = 297.867987566
precise_stop = 297.868767675
reraise    = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
result     = None
start      = 1644094783.546817
stop       = 1644094783.5475986
when       = 'setup'

.tox/python/lib/python3.10/site-packages/_pytest/runner.py:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/python/lib/python3.10/site-packages/_pytest/runner.py:261: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
        ihook      = <_HookCaller 'pytest_runtest_setup'>
        item       = <Function test_ssl_adapters[builtin]>
        kwds       = {}
.tox/python/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
        argname    = 'item'
        args       = ()
        firstresult = False
        kwargs     = {'item': <Function test_ssl_adapters[builtin]>}
        self       = <_HookCaller 'pytest_runtest_setup'>
.tox/python/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
        firstresult = False
        hook_name  = 'pytest_runtest_setup'
        kwargs     = {'item': <Function test_ssl_adapters[builtin]>}
        methods    = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/runner/work/cheroot/cheroot/.tox/python/lib/p...pper name='/dev/null' mode='r' encoding='UTF-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>>, ...]
        self       = <_pytest.config.PytestPluginManager object at 0x7f068b7c6b30>
.tox/python/lib/python3.10/site-packages/_pytest/runner.py:156: in pytest_runtest_setup
    item.session._setupstate.setup(item)
        item       = <Function test_ssl_adapters[builtin]>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.runner.SetupState object at 0x7f068a335930>
item = <Function test_ssl_adapters[builtin]>

    def setup(self, item: Item) -> None:
        """Setup objects along the collector chain to the item."""
        needed_collectors = item.listchain()
    
        # If a collector fails its setup, fail its entire subtree of items.
        # The setup is not retried for each item - the same exception is used.
        for col, (finalizers, exc) in self.stack.items():
>           assert col in needed_collectors, "previous item was not torn down properly"
E           AssertionError: previous item was not torn down properly

col        = <Module test_server.py>
exc        = None
finalizers = [<bound method Node.teardown of <Module test_server.py>>]
item       = <Function test_ssl_adapters[builtin]>
needed_collectors = [<Session cheroot exitstatus=<ExitCode.OK: 0> testsfailed=0 testscollected=157>, <Package test>, <Module test_ssl.py>, <Function test_ssl_adapters[builtin]>]
self       = <_pytest.runner.SetupState object at 0x7f068a335930>

.tox/python/lib/python3.10/site-packages/_pytest/runner.py:489: AssertionError

This is what tox has installed in the env that crashes:

python installed: attrs==21.4.0,certifi==2021.10.8,cffi==1.15.0,chardet==4.0.0,charset-normalizer==2.0.11,cheroot @ file:///home/runner/work/cheroot/cheroot/.tox/.tmp/package/2/cheroot-8.6.1.dev216%2Bgfd10ba99-py2.py3-none-any.whl,colorama==0.4.4,coverage==6.2,cryptography==36.0.1,docopt==0.6.2,execnet==1.9.0,idna==3.3,iniconfig==1.1.1,jaraco.context==4.1.1,jaraco.functools==3.5.0,jaraco.text==3.7.0,more-itertools==8.12.0,packaging==21.3,pluggy==1.0.0,portend==3.1.0,py==1.11.0,pycparser==2.21,pyOpenSSL==22.0.0,pyparsing==3.0.7,pypytools==0.6.2,pytest==7.0.0,pytest-cov==2.12.0,pytest-forked==1.4.0,pytest-mock==3.7.0,pytest-rerunfailures==10.2,pytest-sugar==0.9.4,pytest-watch==4.2.0,pytest-xdist==2.5.0,pytz==2021.3,requests==2.27.1,requests-toolbelt==0.9.1,requests-unixsocket==0.3.0,six==1.16.0,tempora==5.0.1,termcolor==1.1.0,tomli==2.0.0,trustme==0.9.0,urllib3==1.26.8,watchdog==2.1.6

plugins: xdist-2.5.0, forked-1.4.0, mock-3.7.0, sugar-0.9.4, rerunfailures-10.2, cov-2.12.0

I’ve seen this log on GHA under Ubuntu 20.04 and 18.04. Can’t say about other envs because they’ve gotten canceled (will need to check separately).

I don’t have a stable repro, unfortunately, but here’s CI logs: https://github.com/cherrypy/cheroot/runs/5079935409.

[bluetech]

It seems all affected use pytest-forked. So it’s probably that, not rerunfailures.

• cherrypy/cheroot - uses pytest.mark.forked
• @cnsnyder (do you use @pytest.mark.forked in particular?)
• learning-at-home/hivemind - uses @pytest.mark.forked

[The-Compiler]

FWIW no issues so far with my project (qutebrowser) which uses pytest-rerunfailures.