Pip 19.1 pip install --no-use-pep517 not working with nested requirements files

Relevent issues:

https://github.com/pypa/pip/issues/6375 https://github.com/pypa/pip/issues/6314 https://github.com/pypa/pip/pull/6370

Environment

• pip version: 19.1
• Python version: 3.6
• OS: Linux

Description

pip install fails with a requirements file that references a setup.py file, in a project that has a pyproject.toml file that has no build tooling section.

My project setup:

setup.py  # Contains the actual project/package requirements
pyproject.toml  # Contains only 2 lines of black configuration
requirements/base.txt  # Contains only `-e .`
requirements/testing.txt  # Contains `-r base.txt` and then other stuff like pytest etc.
src/mypackage/...  # Actual project/package code

This setup allows us to:

• Specify the project requirements exactly once (in the setup.py)
• Use a command like pip install -r requirements/testing.txt to install both the project requirements along with other stuff needed to run the tests.
• Build the package with python setup.py sdist

The suggested workaround in https://github.com/pypa/pip/pull/6370#issuecomment-486153504 doesn’t seem to work for me:

pip install --no-use-pep517 -q --upgrade --requirement /..../src/requirements/development.txt
ERROR: Error installing 'file:///.... (from -r /..../src/requirements/_base.txt (line 1))': editable mode is not supported for pyproject.toml-style projects. pip is processing this project as pyproject.toml-style because it has a pyproject.toml file. Since the project has a setup.py and the pyproject.toml has no "build-backend" key for the "build_system" value, you may pass --no-use-pep517 to opt out of pyproject.toml-style processing. See PEP 517 for details on pyproject.toml-style projects.

pip install -r requirements/testing.txt fails

Expected behavior

Historically (in <= 19.0.3) pip would install the project and testing/dev requirements (into the current virtualenv).

How to Reproduce

Set up a project as per above, try to install the dev/testing requirements with pip install -r foo.txt.

Output

As per description.

[pitrou]

Asking people to type the obscure --no-use-pep517 (should people even have to know what PEP 517 is?) when the well-known pip install -e used to work is not a sane workaround. I’m astonished that the pip maintainers estimate it’s fine to break well-known practices just out of theoretical purity.

Of course, there’s still python setup.py develop…

[gaborbernat]

So I get now pip is PEP compliant, but this now breaks peoples workflow without any sane workaround, I consider it as a PEP oversight. PEP-517/8 on purpose did not want to cover editable installs, considering something to be addressed down the line. As such I would argue PEP-517/PEP-518 no longer applies when editable install mode is enabled. It’s the build frontend dependent on how this case is handled for now. In such a case, pip should fallback to the old system, until we define editable installs inside a PEP. @pfmoore want to weigh in? If we want people to explicitly require the old behaviour that’s okay, but then --no-use-pep517 should work independently if the pyproject.toml is there or what it contains.