pytest 5.3.3: slow or crashing tests

tl;dr: If your tests are slow or break on pytest 5.3.3, please pin pytest==5.3.2 until this is resolved. A change to fixture handling seems incompatible with at least pytest-django, but also some non-Django projects.

A change in 99180939febb6ca4b3ff788cf336b065046653f2 (part of the 5.3.3 release) led to an incompatibility regarding fixture handling/cleanup. It seems fixtures are not correctly recycled/collected, so they persist in unintended ways.

Original report with plenty of unimportant details inside:

I'm testing a Django project with pytest, pytest-xdist, pytest-django, and tox as runner. As of the latest pytest release ~2 hours ago, 5.3.3, I'm seeing the weirdest test failures.

The errors can be seen here:

django.core.exceptions.ImproperlyConfigured: The database name 'test_travis_ci_test_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0' (71 characters) is longer than PostgreSQL's limit of 63 characters. Supply a shorter NAME in settings.DATABASES.
---------------------------- Captured stderr setup -----------------------------
Got an error creating the test database: database "test_travis_ci_test_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0_gw0" already exists

or using MySQL:

---------------------------------------------------------------------------------------------- Captured stderr setup ----------------------------------------------------------------------------------------------
Got an error creating the test database: (1102, "Incorrect database name 'test_pretalx_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7_gw7'")

The database name is supposedly generated by pytest-xdist, which is running on gw0 and gw1 in this Travis runner.

From my local debugging: It appears that initally, pytest-xdist will create databases called “test_pretalx_gw0” and so on, but then I see those disappear and be replaced by “test_pretalx_gw0_gw0”, then “test_pretalx_gw0_gw0_gw0”, and so on, while the tests are running. They aren’t all replaced at the same time, and not for each test, but (I think) each time each thread opens a new module/test file.

I’m not sure if the fault rests with pytest, or pytest-xdist, or pytest-django, but it’s only present as of the 5.3.3. bugfix release, with 5.3.2 running as intended (and significantly faster, at that). I’m reporting this here in hopes of help, and because while the error implies that the problem lies with either pytest-xdist or pytest-django, the change was definitely introduced on the side of pytest itself.

I’m happy to provide more details or debugging.

</detail>

[mvbrn]

@blueyed pytest-django is present 😃

upd: downgrade to pytest==5.3.2 helped me

[nicoddemus]

Thanks @rixx.

That confirms my suspicion that it is related to #6438.

The fix might have caused a regression in pytest-django though, so I suggest to cross-post this in https://github.com/pytest-dev/pytest-django/issues. 👍