pytest fails for source installs that don't use `pip install --editable .`

Strangely it seems that if we install from source with

python -m pip install .[test]

pytest will fail with things like

==================================== ERRORS ====================================
_____________________ ERROR collecting src/pyhf/compat.py ______________________
/opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/site-packages/_pytest/runner.py:311: in from_call
    result: Optional[TResult] = func()
/opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/site-packages/_pytest/runner.py:341: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/site-packages/_pytest/doctest.py:532: in collect
    module = import_path(self.fspath)
/opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/site-packages/_pytest/pathlib.py:544: in import_path
    raise ImportPathMismatchError(module_name, module_file, path)
E   _pytest.pathlib.ImportPathMismatchError: ('pyhf.compat', '/opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/site-packages/pyhf/compat.py', PosixPath('/home/runner/work/pyhf/pyhf/src/pyhf/compat.py'))

and that it will only pass if we install from source in editable mode

python -m pip install --editable .[test]

_Originally posted by @matthewfeickert in https://github.com/scikit-hep/pyhf/issues/1453#issuecomment-841541604_

It would be good to understand why this is, and if possible change it. The tests needing things to be installed with pip install --editable . seems strange and bad.

[diazona]

I spent an hour or so digging into this and I found some useful info. pytest pulls in tests/conftest.py as part of its first pass, before it loads any other test files or source code, and in particular, before it adds src/ to sys.path for doctest discovery. Since conftest.py imports pyhf, it will load the versions of the modules from the installed version of the package, i.e. lib/pythonX.Y/site-packages/pyhf/. Then later, when pytest goes looking for (doc)tests under src/, it prepends src/ to sys.path (NB for doctests, it always prepends, ignoring the --import-mode CLI option) in an effort to import the .py files from within src/, but those modules already exist in sys.modules, having been imported from the installed version. The conflict between the already-imported module and the file that pytest is trying to newly import seems to cause the mismatch error.

So ultimately, if you can remove the dependency of tests/conftest.py on the pyhf code, that should do the trick. I did a quick “feasibility study” by removing the whole parameter list of backend() and the pyhf import, and it seems to allow things to proceed past test collection, which is a good sign, though I did get a test to fail so maybe there are other things to fix.

[matthewfeickert]

I’m not sure (need to test) but I think it is because we have src in our testpaths in pyproject.toml

https://github.com/scikit-hep/pyhf/blob/546022254a526ea62af8cbc6594ec881cddfa9f1/pyproject.toml#L41-L44