pip still failing to install packages with setup.py and pyproject.toml without cache and downloaded offline

Description

Today I ran into an issue where I could not install a Python package that was downloaded with pip download without network access (or using the package cache), and I came across similar issues discussing this corner case, like https://github.com/pypa/pip/issues/7874 and https://github.com/sqlalchemy/alembic/issues/672; in particular, #7874 says:

This is almost certainly a bug in whatever debian does to pip when it packages it up, not a bug in pip itself. (Originally posted by @pganssle in https://github.com/pypa/pip/issues/7874#issuecomment-601855349_)

Well, I am encountering this bug in Centos 7.9 and RHEL 7.9 running Python 3.8, and with macOS 10.14 running Python 3.9, installing a totally different package (in this case, https://github.com/ListenNotes/podcast-api-python). Not Debian. And either using the version of pip that ships with these versions of Python, or using the very latest version of pip. So, are you absolutely sure this isn’t an issue with how pip handles offline (and cache-less) installation of packages that use pyproject.toml? (I don’t think offline/cache-less behaviour is tested as much as online behaviour is – it’s a common assumption with Internet software that all computers are always connected reliably to the Internet, after all.)

Expected behavior

No response

pip version

21.3.1

Python version

3.9

OS

macOS 10.14.6

How to Reproduce

First, I download the package (on a machine with network access, in a new virtual environment):

(tmp-58655d62ae0a345c) $ pip download podcast-api==1.1.0 -d python-packages --no-cache
Collecting podcast-api==1.1.0
  Downloading podcast-api-1.1.0.zip (393 kB)
     |████████████████████████████████| 393 kB 10.8 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Saved ./python-packages/podcast-api-1.1.0.zip
Collecting requests>=2.20; python_version >= "3.0"
  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 13.2 MB/s 
  Saved ./python-packages/requests-2.26.0-py2.py3-none-any.whl
Collecting setuptools>=41.0.1
  Downloading setuptools-59.3.0-py3-none-any.whl (952 kB)
     |████████████████████████████████| 952 kB 11.5 MB/s 
  Saved ./python-packages/setuptools-59.3.0-py3-none-any.whl
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
     |████████████████████████████████| 138 kB 11.1 MB/s 
  Saved ./python-packages/urllib3-1.26.7-py2.py3-none-any.whl
Collecting charset-normalizer~=2.0.0; python_version >= "3"
  Downloading charset_normalizer-2.0.8-py3-none-any.whl (39 kB)
  Saved ./python-packages/charset_normalizer-2.0.8-py3-none-any.whl
Collecting certifi>=2017.4.17
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
     |████████████████████████████████| 149 kB 11.0 MB/s 
  Saved ./python-packages/certifi-2021.10.8-py2.py3-none-any.whl
Collecting idna<4,>=2.5; python_version >= "3"
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 12.7 MB/s 
  Saved ./python-packages/idna-3.3-py3-none-any.whl
Successfully downloaded podcast-api requests setuptools urllib3 charset-normalizer certifi idna

This results in these packages downloaded:

(tmp-58655d62ae0a345c) $ ls -1 python-packages/
certifi-2021.10.8-py2.py3-none-any.whl
charset_normalizer-2.0.8-py3-none-any.whl
idna-3.3-py3-none-any.whl
podcast-api-1.1.0.zip
requests-2.26.0-py2.py3-none-any.whl
setuptools-59.3.0-py3-none-any.whl
urllib3-1.26.7-py2.py3-none-any.whl

I note that setuptools did get downloaded; I’m not exactly sure why. The command was run with Python 3.9 which ships with 46.1.3, and the podcast-api package has a pyproject.toml with:

[build-system]
requires = [
    'setuptools>=40.8.0', 'wheel'
]

and a setup.py with:

    install_requires=[
        'requests >= 2.20; python_version >= "3.0"',
        "setuptools>=41.0.1",
    ],

Ignoring the slight version mismatch there, is pip downloading setuptools because it’s been explicitly been identified in the install_requires section?

If I install the package from what got downloaded (some parts snipped for brevity), first it finds podcast-api, downloaded as a .zip:

(tmp-58655d62ae0a345c) $ pip3 install -v podcast-api==1.1.0 --find-links python-packages --no-deps --no-index --no-cache
Non-user install because user site-packages disabled
Ignoring indexes: https://pypi.org/simple
...
Looking in links: python-packages
0 location(s) to search for versions of podcast-api:
Skipping link: not a file: python-packages (from -f)
Skipping link: wrong project name (not podcast-api): file:///private/tmp/python-packages/urllib3-1.26.7-py2.py3-none-any.whl
Found link file:///private/tmp/python-packages/podcast-api-1.1.0.zip, version: 1.1.0
Skipping link: wrong project name (not podcast-api): file:///private/tmp/python-packages/setuptools-59.3.0-py3-none-any.whl
Skipping link: wrong project name (not podcast-api): file:///private/tmp/python-packages/charset_normalizer-2.0.8-py3-none-any.whl
Skipping link: wrong project name (not podcast-api): file:///private/tmp/python-packages/requests-2.26.0-py2.py3-none-any.whl
Skipping link: wrong project name (not podcast-api): file:///private/tmp/python-packages/idna-3.3-py3-none-any.whl
Skipping link: wrong project name (not podcast-api): file:///private/tmp/python-packages/certifi-2021.10.8-py2.py3-none-any.whl
Local files found: /private/tmp/python-packages/podcast-api-1.1.0.zip
Given no hashes to check 1 links for project 'podcast-api': discarding no candidates
Using version 1.1.0 (newest of versions: 1.1.0)
Processing ./python-packages/podcast-api-1.1.0.zip
  Added podcast-api==1.1.0 from file:///private/tmp/python-packages/podcast-api-1.1.0.zip to build tracker '/private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-req-tracker-nd04s5ue'
  Created temporary directory: /private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-build-env-g9znb5ko
  Running command /Users/tom/venvs/tmp-58655d62ae0a345c/bin/python /Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-build-env-g9znb5ko/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: --no-index --find-links python-packages -- 'setuptools>=40.8.0' wheel

And then decides it needs to install setuptools, which it finds downloaded as a .whl:

  Non-user install by explicit request
  Ignoring indexes: https://pypi.org/simple
  ...
  Looking in links: python-packages
  0 location(s) to search for versions of setuptools:
  Skipping link: not a file: python-packages (from -f)
  Skipping link: wrong project name (not setuptools): file:///private/tmp/python-packages/urllib3-1.26.7-py2.py3-none-any.whl
  Skipping link: Missing project version for setuptools: file:///private/tmp/python-packages/podcast-api-1.1.0.zip
  Found link file:///private/tmp/python-packages/setuptools-59.3.0-py3-none-any.whl, version: 59.3.0
  Skipping link: wrong project name (not setuptools): file:///private/tmp/python-packages/charset_normalizer-2.0.8-py3-none-any.whl
  Skipping link: wrong project name (not setuptools): file:///private/tmp/python-packages/requests-2.26.0-py2.py3-none-any.whl
  Skipping link: wrong project name (not setuptools): file:///private/tmp/python-packages/idna-3.3-py3-none-any.whl
  Skipping link: wrong project name (not setuptools): file:///private/tmp/python-packages/certifi-2021.10.8-py2.py3-none-any.whl
  Local files found: /private/tmp/python-packages/setuptools-59.3.0-py3-none-any.whl
  Given no hashes to check 1 links for project 'setuptools': discarding no candidates
  Using version 59.3.0 (newest of versions: 59.3.0)
  Processing ./python-packages/setuptools-59.3.0-py3-none-any.whl
    Added setuptools>=40.8.0 from file:///private/tmp/python-packages/setuptools-59.3.0-py3-none-any.whl to build tracker '/private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-req-tracker-nd04s5ue'
    Removed setuptools>=40.8.0 from file:///private/tmp/python-packages/setuptools-59.3.0-py3-none-any.whl from build tracker '/private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-req-tracker-nd04s5ue'

And finally it decides it also needs wheel, which it doesn’t find:

  0 location(s) to search for versions of wheel:
  Skipping link: wrong project name (not wheel): file:///private/tmp/python-packages/setuptools-59.3.0-py3-none-any.whl
  Given no hashes to check 0 links for project 'wheel': discarding no candidates
  ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
  ERROR: No matching distribution found for wheel
  Exception information:
  Traceback (most recent call last):
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
      status = self.run(options, args)
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
      return func(self, options, args)
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 332, in run
      requirement_set = resolver.resolve(
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 313, in _get_abstract_dist_for
      self._populate_link(req)
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 279, in _populate_link
      req.link = self.finder.find_requirement(req, upgrade)
    File "/Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip/_internal/index/package_finder.py", line 928, in find_requirement
      raise DistributionNotFound(
  pip._internal.exceptions.DistributionNotFound: No matching distribution found for wheel
  Removed build tracker: '/private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-req-tracker-nd04s5ue'
  Installing build dependencies ... error
...
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/tom/venvs/tmp-58655d62ae0a345c/bin/python /Users/tom/venvs/tmp-58655d62ae0a345c/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-build-env-g9znb5ko/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: --no-index --find-links python-packages -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
Removed podcast-api==1.1.0 from file:///private/tmp/python-packages/podcast-api-1.1.0.zip from build tracker '/private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-req-tracker-nd04s5ue'
Removed build tracker: '/private/var/folders/13/4gh6hy888xjgdq001s6kn2vr0000gn/T/pip-req-tracker-nd04s5ue'

If I manually download the wheel for wheel (heh) with pip wheel wheel -w python-packages, and then run the installation for podcast-api, it succeeds, because now it can find a wheel for wheel.

Here are my questions:

Why doesn’t pip use the version of wheel that exists in the Python installation on disk, when building the wheel for the package? Is it because wheel is not specified in install_requires?

But even if wheel is specified in install_requires, won’t it likely already be present in any given Python installation, and therefore not downloaded/installed?

But even if a super-new version of wheel was specified, like 0.37.0, so the version in the Python installation is older, won’t that only work until the installation is done on a system that has that version of wheel already (e.g. Python 3.10 or 3.11 or whatever)?

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[shaggyfrog]

With all due respect, it’s not acceptable to demand solutions from people giving their time freely to provide software that you can use with no cost or limitations.

That’s not true, I have not demanded a solution from anyone, and certainly not one “with no cost or limitations”. At most, I’ve asked to have a consensus that this is, in fact, a bug in pip. Uncool.

[pfmoore]

With all due respect, that is just not an acceptable solution to propose to an end-user

With all due respect, it’s not acceptable to demand solutions from people giving their time freely to provide software that you can use with no cost or limitations.

And as regards my suggestion, it was just that - a suggestion. I don’t see how offering a suggestion is unacceptable. You’re completely entitled to say that it’s not useful to you and leave it at that.

If a project uses pyproject.toml, it has chosen to use a new standard that works differently than the old setuptools-only build process. What you are seeing may be a bug. It may be a “missing feature” that no-one considered when designing the new standard. Or it may simply be a design choice that doesn’t suit your use case. I don’t personally intend to spend my free time investigating which it is, but hopefully you will be able to find someone willing to help you out by doing so.

Best of luck, and (genuinely!) thank you for spending your time contributing a bug report with the results of your investigations.