Unable to build no-binary :all: with use-pep517

Environment

• pip version: 20.0.2
• Python version: 3.8.1
• OS: macOS 10.15.3

Description

As distlib doesn’t publish a wheel, and because it has an implicit dependency on setuptools, I’m attempting to use --use-pep517 to force pip to ensure that setuptools/wheel are present for building, but when I do, pip fails to install wheel:

~ $ pip install --use-pep517 --no-binary :all: distlib                                                                                                                        
Collecting distlib
  Using cached distlib-0.3.0.zip (571 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-6j9zlufg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (13 lines):
  Collecting setuptools>=40.8.0
    Using cached setuptools-45.3.0.zip (862 kB)
  Collecting wheel
    Using cached wheel-0.34.2.tar.gz (58 kB)
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/pip-egg-info
           cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-6j9zlufg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-mzykfthi', '--use-pep517', '--no-binary', ':all:', 'distlib')' returned non-zero exit status 1.

Expected behavior

pip should be able to assemble a build environment, either by disabling the no-binary option for those requirements or (preferably) by providing a bootstrap environment for these build dependencies.

Note: This may be an issue that can be addressed with wheel.

[pfmoore]

Is this not the situation that in-tree build backends were designed to address? Does setuptools configure itself as its own in-tree build backend? If not, then maybe it needs to for the --no-binaty :all: case.

[jaraco]

I ran into this issue again today, this time with pip download. I was attempting to download the sources for xattr:

pip download --no-binary :all: xattr

Because I’m trying to be a good citizen and operate without setuptools being installed, that operation failed with:

ImportError: cannot import name ‘setup’ from ‘setuptools’ (unknown location)

When setuptools tries to build from source, which tries to build wheel from source, and setuptools isn’t present.

 draft $ py -m pip download --no-binary :all: xattr
Collecting xattr
  Downloading xattr-0.10.0.tar.gz (15 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
      Collecting setuptools>=40.8.0
        Using cached setuptools-65.5.0.tar.gz (2.6 MB)
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'error'
        error: subprocess-exited-with-error
      
        × pip subprocess to install backend dependencies did not run successfully.
        │ exit code: 1
        ╰─> [26 lines of output]
            DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
            Collecting wheel
              Using cached wheel-0.37.1.tar.gz (66 kB)
              Preparing metadata (setup.py): started
              Preparing metadata (setup.py): finished with status 'error'
              error: subprocess-exited-with-error
      
              × python setup.py egg_info did not run successfully.
              │ exit code: 1
              ╰─> [6 lines of output]
                  Traceback (most recent call last):
                    File "<string>", line 2, in <module>
                    File "<pip-setuptools-caller>", line 34, in <module>
                    File "/private/var/folders/sx/n5gkrgfx6zd91ymxr2sr9wvw00n8zm/T/pip-install-eca5mryf/wheel_44b7c370255345dd8b9fc33a8f13d4d7/setup.py", line 2, in <module>
                      from setuptools import setup
                  ImportError: cannot import name 'setup' from 'setuptools' (unknown location)
                  [end of output]
      
              note: This error originates from a subprocess, and is likely not a problem with pip.
            error: metadata-generation-failed
      
            × Encountered error while generating package metadata.
            ╰─> See above for output.
      
            note: This is an issue with the package mentioned above, not pip.
            hint: See above for details.
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error
      
      × pip subprocess to install backend dependencies did not run successfully.
      │ exit code: 1
      ╰─> See above for output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I tried instead to employ pep517, but that failed similarly:

 draft $ py -m pip download --use-pep517 --no-binary :all: xattr
Collecting xattr
  Using cached xattr-0.10.0.tar.gz (15 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
      Collecting setuptools>=40.8.0
        Using cached setuptools-65.5.0.tar.gz (2.6 MB)
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'error'
        error: subprocess-exited-with-error
      
        × pip subprocess to install backend dependencies did not run successfully.
        │ exit code: 1
        ╰─> [26 lines of output]
            DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
            Collecting wheel
              Using cached wheel-0.37.1.tar.gz (66 kB)
              Preparing metadata (setup.py): started
              Preparing metadata (setup.py): finished with status 'error'
              error: subprocess-exited-with-error
      
              × python setup.py egg_info did not run successfully.
              │ exit code: 1
              ╰─> [6 lines of output]
                  Traceback (most recent call last):
                    File "<string>", line 2, in <module>
                    File "<pip-setuptools-caller>", line 34, in <module>
                    File "/private/var/folders/sx/n5gkrgfx6zd91ymxr2sr9wvw00n8zm/T/pip-install-_c2chljg/wheel_1e5d23c0a353431cac90e861087cc190/setup.py", line 2, in <module>
                      from setuptools import setup
                  ImportError: cannot import name 'setup' from 'setuptools' (unknown location)
                  [end of output]
      
              note: This error originates from a subprocess, and is likely not a problem with pip.
            error: metadata-generation-failed
      
            × Encountered error while generating package metadata.
            ╰─> See above for output.
      
            note: This is an issue with the package mentioned above, not pip.
            hint: See above for details.
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error
      
      × pip subprocess to install backend dependencies did not run successfully.
      │ exit code: 1
      ╰─> See above for output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

The only way I know to make it work is to pre-install setuptools (works with or without --use-pep517). I was surprised that --use-pep517 succeeded.

 draft $ py -m pip-run -q setuptools -- -m pip download --use-pep517 --no-binary :all: xattr
Collecting xattr
  Using cached xattr-0.10.0.tar.gz (15 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.0
  Using cached cffi-1.15.1.tar.gz (508 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pycparser
  Using cached pycparser-2.21.tar.gz (170 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Saved ./xattr-0.10.0.tar.gz
Saved ./cffi-1.15.1.tar.gz
Saved ./pycparser-2.21.tar.gz
Successfully downloaded xattr cffi pycparser