Build dependencies (PEP 518) not respected by `pip download`

Environment

• pip version: 20.0.2
• Python version: 3.6
• OS: Ubuntu 18.04

Description An issue was reported in my repo https://github.com/pydata/bottleneck/issues/333 by a user utilizing pip download to create a local cache prior to installation, but was seeing the following error despite having an up-to-date copy of setuptools:

  ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
  ERROR: No matching distribution found for setuptools

I was able to reproduce the behavior via a Dockerfile and identify that the root issue is that pip download is not fetching the PEP 517 dependencies. This error message appears to be reproducible for other packages (such as numpy) provided that pip download fetches a source package that requires a PEP 517 build step.

My findings can be found here: https://github.com/pydata/bottleneck/issues/333#issuecomment-599177688

Expected behavior

As far as expected behavior, I would expect pip download to fetch all packages needed to successfully install the target package(s), including build-only dependencies.

How to Reproduce Dockerfile:

FROM nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04
RUN apt-get update
RUN apt-get install -y python3 python3-pip
RUN pip3 install -U pip
RUN cd /tmp && python3 -m pip download numpy --no-binary numpy -d ./
RUN ls -alh /tmp
RUN python3 -m pip install --no-binary numpy numpy --find-links /tmp --no-index
RUN pip3 list

Output:

Step 7/8 : RUN python3 -m pip install --no-binary numpy numpy --find-links /tmp --no-index
 ---> Running in b34d3e92b764
Looking in links: /tmp
Processing /tmp/numpy-1.18.1.zip
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-6p0zdywu/overlay --no-warn-script-location --no-binary numpy --only-binary :none: --no-index --find-links /tmp -- setuptools wheel 'Cython>=0.29.14'
       cwd: None
  Complete output (3 lines):
  Looking in links: /tmp
  ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
  ERROR: No matching distribution found for setuptools
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-6p0zdywu/overlay --no-warn-script-location --no-binary numpy --only-binary :none: --no-index --find-links /tmp -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output.
The command '/bin/sh -c python3 -m pip install --no-binary numpy numpy --find-links /tmp --no-index' returned a non-zero code: 1

[pganssle]

@pradyunsg I think downloading the build dependencies of anything that uses PEP 518 and isn’t satisfied by a wheel would be a reasonable choice. If the main idea is that pip download is used for caching everything you need to build and install the projects you’re downloading, then the build-time dependencies need to be satisfied as well if they’re needed.

(Obviously this won’t work for projects that don’t use PEP 518, but that seems like a good incentive for those projects to opt in.)

[pradyunsg]

Thanks for a super detailed bug report @qwhelan!

pip download does not store the build dependencies for the package; which IIUC, is what this issue is about.

I’m personally not sure about how pip download should handle build dependencies, so I’ll think a bit more about this before coming back.