Presence of a pyproject.toml file causing unexpected behavior

pip version

21.0.1

Python version

3.8.0

OS

Ubuntu 18.04

Additional information

No response

Description

Somehow, the presence of a pyproject.toml file is causing pip install from a directory to fail importing of installed modules.

Expected behavior

The presence of pyproject.toml does not impact the behavior of pip install.

How to Reproduce

This is a bit complicated to explain minimal repro here should be enough.

Structure:

$ tree
.
├── build_helper
│   ├── build_helper
│   │   └── __init__.py
│   └── setup.py
├── example
│   └── __init__.py
├── README.md
└── setup.py

Content of files in repro repository.

Output

Installing build_helper:

$ pip install build_helper/
Processing ./build_helper
Building wheels for collected packages: build-helper
  Building wheel for build-helper (setup.py) ... done
  Created wheel for build-helper: filename=build_helper-0.0.0-py3-none-any.whl size=1261 sha256=aa0ee67ef3385cba13e32659fb3635047631d0466ef0861c71badd5d3d67f7fd
  Stored in directory: /tmp/pip-ephem-wheel-cache-_dpz2jwm/wheels/36/26/ae/14e3035420d431e16878ed266d823b24627dc0fe56af0b53b3
Successfully built build-helper
Installing collected packages: build-helper
  Attempting uninstall: build-helper
    Found existing installation: build-helper 0.0.0
    Uninstalling build-helper-0.0.0:
      Successfully uninstalled build-helper-0.0.0
Successfully installed build-helper-0.0.0

Successfully installing primary project:

$ pip install .
Processing /home/omry/dev/test_pip
Building wheels for collected packages: example
  Building wheel for example (setup.py) ... done
  Created wheel for example: filename=example-1.0.3-py3-none-any.whl size=991 sha256=21a95c7e50946865cb6f4565842e455ab9a972ceae1800e534d391e2032286c0
  Stored in directory: /tmp/pip-ephem-wheel-cache-k02av0q_/wheels/57/c2/45/2447e9fc2acaa90e70563527f393553ccc1297fb8d529f0c92
Successfully built example
Installing collected packages: example
  Attempting uninstall: example
    Found existing installation: example 1.0.3
    Uninstalling example-1.0.3:
      Successfully uninstalled example-1.0.3
Successfully installed example-1.0.3

Creating an empty pyproject.yaml, after which installation fails:

$ touch pyproject.toml
$ pip install .
Processing /home/omry/dev/test_pip
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/omry/miniconda3/envs/test-pip/bin/python /home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpybcsa03y
       cwd: /tmp/pip-req-build-_31oxf1l
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 258, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 3, in <module>
      from build_helper import get_version
  ImportError: cannot import name 'get_version' from 'build_helper' (unknown location)
  ----------------------------------------
WARNING: Discarding file:///home/omry/dev/test_pip. Command errored out with exit status 1: /home/omry/miniconda3/envs/test-pip/bin/python /home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpybcsa03y Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/omry/miniconda3/envs/test-pip/bin/python /home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpybcsa03y Check the logs for full command output.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[omry]

I am closing this, thanks for the discussion and help.

Here is a short summary (happy to update if I missed anything important):

• pip switching on isolated builds based on the presence of a pyproject.toml is controversial but intentional. There is an in-depth discussion at #8437.
• My underlying issue can be solved by publishing the build helpers or by enabling in-tree build (–use-feature=in-tree-build). In-tree build are planned to become the default.

[pfmoore]

However, for some project it makes sense to consider the whole checkout tree, even if the installation is of a sub directory.

That is in violation of PEP 517, which views the directory with setup.py/pyproject.toml as the source tree. Therefore, in your example, build-helper is helper code outside the source tree, which violates the principle that sources should be self-contained.

Let me ask this. If you built a sdist for this project, what would it contain? Assume for now that you do setup.py sdist, so pip isn’t involved in that question.

I couldn’t figure out how to pass config for pyproject.toml to it.

You don’t need to, that file is (by definition) at the root of the source tree, so it’s in your CWD (build hooks are called with the source directory as CWD).

I suspect that the rest of the source tree is not yet available in the isolated dir when the tree is running.

I suspect that you’re meaning “files outside the source tree” when you say “rest” (if we accept that the source tree is rooted in the directory with pyproject.toml as PEP 517 requires).

Here are the setup.py files of each I could along with my untested judgement if they will have issues or not to transfer to a pure data driven build system

No one is suggesting a “pure data driven build system”. Setuptools is, and probably always will be, a procedural build system. What we’re trying to do is to make more of the build metadata and configuration that is common to all build backends, available without needing to run the project build process. But the build step itself can do whatever it likes.

The root cause of this particular issue is that one piece of data that front ends want to know is “how do I set up a fresh environment which I can use to run the build steps for this project?” Your examples have fundamentally all been of the form “you need to install package X, but package X isn’t publicly available”.

Hmm, putting the question this way makes me think. In your very original example:

$ tree
.
├── build_helper
│   ├── build_helper
│   │   └── __init__.py
│   └── setup.py
├── example
│   └── __init__.py
├── README.md
└── setup.py

why can’t setup.py simply do

import sys
from pathlib import Path
sys.path.append(Path(__file__).parent / "build_helper")

There’s no need to actually install build-helper in this case - it’s present in the source tree (and presumably shipped in the sdist) so just use it directly.

I assume there’s probably some other constraint that you didn’t mention, but I’m not clear what that is.