conftest.py files in packages matching "test*" are being loaded, even if they're not in the `testpaths` list

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

I am defining testpaths = tests in pytest.ini and that’s being picked up by pytest (see pytest output below), but then it’s loading a conftest.py from another directory.

Reproduction steps

Environment setup

% uname -a
Linux m 5.15.12-arch1-1 #1 SMP PREEMPT Wed, 29 Dec 2021 12:04:56 +0000 x86_64 GNU/Linux
% python --version
Python 3.10.1
% python -m venv ../venv
% ../venv/bin/pip install pytest
Collecting pytest
Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting attrs>=19.2.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting iniconfig
Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting toml
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting py>=1.8.2
Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting pluggy<2.0,>=0.12
Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.6-py3-none-any.whl (97 kB)
Installing collected packages: pyparsing, toml, py, pluggy, packaging, iniconfig, attrs, pytest
Successfully installed attrs-21.4.0 iniconfig-1.1.1 packaging-21.3 pluggy-1.0.0 py-1.11.0 pyparsing-3.0.6 pytest-6.2.5 toml-0.10.2
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '[snip]/venv/bin/python -m pip install --upgrade pip' command.
% ../venv/bin/pip list
Package    Version
---------- -------
attrs      21.4.0
iniconfig  1.1.1
packaging  21.3
pip        21.2.4
pluggy     1.0.0
py         1.11.0
pyparsing  3.0.6
pytest     6.2.5
setuptools 58.1.0
toml       0.10.2
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '[snip]/venv/bin/python -m pip install --upgrade pip' command.

The test files:

% tree
.
├── pytest.ini
├── tests
│   ├── __init__.py
│   └── test_.py
└── testsinvalid
    └── conftest.py

2 directories, 4 files

Their contents:

% find -type f | xargs -IF sh -c "echo; echo === F; cat F"

=== ./pytest.ini
[pytest]
testpaths = tests

=== ./testsinvalid/conftest.py
raise Exception('loading excluded conftest')

=== ./tests/test_.py
def test_p():
    assert True

=== ./tests/__init__.py

The command:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ../venv/bin/pytest

What I expect to see

% PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ../venv/bin/pytest
=================================================================== test session starts ====================================================================
platform linux -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /[snip], configfile: pytest.ini, testpaths: tests
collected 1 item

tests/test_.py .                                                                                                                                     [100%]

==================================================================== 1 passed in 0.00s =====================================================================

What I actually see

% PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ../venv/bin/pytest
ImportError while loading conftest '[snip]/testsinvalid/conftest.py'.
testsinvalid/conftest.py:1: in <module>
raise Exception('loading excluded conftest')
E   Exception: loading excluded conftest

Of note

If I mv testsinvalid anything-without-the-prefix-tests, the problem goes away.

[dhensen]

This has been added in d632a0d, and while I agree it is surprising, it is a documented feature:

if exists, load conftest.py and test*/conftest.py relative to the directory part of the first test path.

So unless we want to somehow change this (including a deprecation period), we can close this as “worked as intended”.

Is this really expected behaviour? I’m in doubt so I’ll share my setup using pytest 7.0.0:

├── lambda_layer
│   ├── layer.zip
│   └── python
│       ├── numpy
│       │   └── conftest.py     <--
├── pytest.ini
├── src
│   ├── mycodehere.py
├── tests
│   └── test_my_test_code_here.py

pytest.ini looks like this:

[tool:pytest]
addopts = -m 'not e2e'
testpaths = tests

When I execute pytest it fails, because numpy’s conftest wants to import dependencies that I don’t have installed. (I don’t want to install those dependencies, I’m making a lambda layer, so those files are maybe there or maybe not, but it feels like it should not load all conftests under my repo root when I have specified testpaths) Is this really expected behaviour?

if exists, load conftest.py and test*/conftest.py relative to the directory part of the first test path

Isn’t my first test path tests as specified via testpaths in my pytest.ini? I guess when reading the manual regarding testpaths it does not really say anything about conftest loading behaviour. What to do with this?

[nicoddemus]

I’m not sure why this is a feature though because I would expect running pytest tests/ and using testpaths = tests when running pytest without any arguments to have the same behaviour, I thought that was the point of testpaths.

I agree, seems like that feature was introduced prior to testpaths.

I guess it is possible to deprecate the behavior: we can detect a conftest.py was matched via test*/conftest.py, and issue a deprecation warning.