Install install_requires before running python setup.py bdist_wheel

Environment

• pip version: 19.0.1
• Python version: CPython 3.7

Description

pip executes python setup.py bdist_wheel before installing the packages listed in install_requires.

For a simple package depending on numpy, using numpy in setup.py but such that python setup.py egg_info works fine without numpy, it shouldn’t be necessary to use setup_requires (and to have a setup.py compatible with setup_requires, which is not easy).

From https://pip.pypa.io/en/latest/reference/pip_install/#installation-order

Although the new install order is not intended to replace (and does not replace) the use of setup_requires to declare build dependencies, it may help certain projects install from sdist (that might previously fail) that fit the following profile:

• They have build dependencies that are also declared as install dependencies using install_requires.
• python setup.py egg_info works without their build dependencies being installed.
• For whatever reason, they don’t or won’t declare their build dependencies using setup_requires.

For example, for this setup.py

import sys
from setuptools import setup

if "egg_info" in sys.argv:
    setup()
    sys.exit()

import numpy as np

setup()

and this setup.cfg:

[metadata]
name = irequires

[options]
install_requires =
    numpy

the command pip install . fails with

Processing /home/users/augier3pi/Dev/try_py_dependencies/irequires
Collecting numpy (from irequires==0.0.0)
  Using cached https://files.pythonhosted.org/packages/3d/10/62224c551acfd3a3583ad16d1e0f1c9e9c333e74479dc51977c31836119c/numpy-1.16.0-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: irequires
  Building wheel for irequires (setup.py) ... error
  Complete output from command /home/users/augier3pi/tmp/tmp_venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-_0f8h24y/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-6n4locuo --python-tag cp37:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-_0f8h24y/setup.py", line 10, in <module>
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
  
  ----------------------------------------
  Failed building wheel for irequires
  Running setup.py clean for irequires
  Complete output from command /home/users/augier3pi/tmp/tmp_venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-_0f8h24y/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-_0f8h24y/setup.py", line 10, in <module>
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
  
  ----------------------------------------
  Failed cleaning build dir for irequires
Failed to build irequires
Installing collected packages: numpy, irequires
  Running setup.py install for irequires ... done
Successfully installed irequires-0.0.0 numpy-1.16.0

Expected behavior

pip should install numpy before executing python setup.py bdist_wheel so that this simple setup works fine as explained in https://pip.pypa.io/en/latest/reference/pip_install/#installation-order.

Note

Listing packages both in install_requires and setup_requires don’t work well. See https://github.com/pypa/setuptools/issues/209 and https://github.com/pypa/setuptools/issues/391

[pradyunsg]

A clarification has been added to the relevant section (see #6604).

Closing this since there isn’t much else actionable here since we mostly just needed a clarification here.

[jdemeyer]

@cjerdonek: The problem is not so much PEP 517, the problem is really bdist_wheel. With PEP 517, bdist_wheel is mandatory and the build fails if the build-and-run-time dependencies are not specified in either pyproject.toml or setup_requires. Without PEP 517, the installation is first done with bdist_wheel, which fails (just like with PEP 517). But in this case, pip falls back to the legacy install mode which does succeed.

So nothing got broken for now (in the sense that packages that used to install still install). But that’s only because pip has a fallback to the old way of installing packages. If that fallback ever gets removed or pyproject.toml becomes mandatory, then this will actually break.