Version 7.1.0 seems like not picking up conftest.py

Over SciPy we have started to see the message in the title in our Azure Pipelines which started picking up 7.1.0 instead of 7.0.1 lastn ight. From the deprecation and removal list we couldn’t see which item might have caused it.

One speculation we might generate is that our slow mark is not registered in our pytest.ini file explicitly and maybe registrations are now obligatory.

Here is a complete SciPy test suite example with all installation steps provided.

https://dev.azure.com/scipy-org/SciPy/_build/results?buildId=16988&view=logs&jobId=7d84eb14-5b07-5555-8e62-838a7038a2b0&j=ede89041-5e8e-5201-ce5c-f68c2cbf02dd&t=141d9d53-a127-58b4-0262-4b9693c8e91f

And our pytest.ini file lives here

https://github.com/scipy/scipy/blob/main/pytest.ini

Thank you for all the efforts.

[bluetech]

This is released in 7.1.1. Thanks @ilayn, @andyfaff, @adrianeboyd, @Akasurde and @mariolenz for the reports, and @The-Compiler for the help.

[bluetech]

OK that makes it clear…

The previous is_in_confcutdir check was:

path not in self._confcutdir.parents

This excludes paths which are directly above confcutdir, e.g. if confcutdir is /a/b/c/, it will exclude /, /a/, /a/b/. But it doesn’t exclude /x/, /a/b/x etc.

The new is_in_confcutdir check is:

try:
    path.relative_to(self._confcutdir)
except ValueError:
    return False
return True

which rejects everything not under confcutdir, e.g. /x etc. are rejected.

The documentation of --confcutdir says only load conftest.py's relative to specified dir. which makes the new behavior sound correct. However, the name “cut” itself as well as c000955dde3ecc12291c8890ba29887d7b6ef1f2 make it sound like the previous behavior was actually the intended one.

The pytest thinc invocation runs the tests out-of-tree - not running against the tests in the source code, but using --pyargs which picks the tests from the venv/site-packages. This is not under the confcutdir, so the new behavior now ignores the conftests there. This is definitely a bug.

However the old behavior is pretty weird as well - e.g. consider a setup like this:

/home/ran/src/thinc/pytest.ini
/home/ran/python/venvs/venv/lib/python3.10/site-packages/thinc/

And we’re running /home/ran/python/venvs/venv/bin/python -m pytest --pyargs thinc.tests. Then pytest will pick up conftests in each of

/home/ran/python/venvs/venv/lib/python3.10/site-packages/
/home/ran/python/venvs/venv/lib/python3.10/
/home/ran/python/venvs/venv/lib/
/home/ran/python/venvs/venv/
/home/ran/python/venvs/
/home/ran/python/

which is definitely not expected.

In any case, for now I’ll revert to the previous behavior (hopefully I can create a test to ensure it doesn’t regress), but will open a follow up issue about the above.