Wheel cache is not forward/backward compatible between 22.2.x and 22.3

Description

Installing packages and having the resulting wheels go into a cache dir with pip 22.3 produces a cache that appears to be ignored by 22.2.2 forcing 22.2.2 to install from sdists. The same thing seems to happen if you create the wheel cache with 22.2.2 and then try to install packages with that cache using 22.3.

We noticed this because we’ve got some multistage docker image builds that will install all build requirements (like gcc) and create wheels for all deps in a throwaway image then copy only the wheels over to the final image and pip install what we need there using the wheel cache to avoid needing build requirements like gcc. This broke when the pip versions on either side of that got out of sync.

I don’t know if this was an expected change (it hasn’t been a problem in the past and I strongly suspect that we’ve gotten out of sync previously). This was also not called out in the release notes. Just want to make sure this gets reported even if ultimately this was expected.

Expected behavior

I expected that the cache would be forward compatible at least. It seems a little odd to force users to rebuild all new wheels when they update pip versions. Old pip recognizing newer pips wheel cache and using it is maybe a bit less expected (though would’ve been nice in our situation).

That said I’m not entirely sure there is a bug here. I’m mostly interested in calling the behavior change out to double check that this was expected. And to see if this sort of behavior change is worthy of a release note.

pip version

22.3

Python version

3.10

OS

Debian Bullseye, OpenSUSE Tumbleweed

How to Reproduce

A working cache with the same pip version across environments:

> python3 -m venv venv1
> venv1/bin/python3 --version
Python 3.10.7
> venv1/bin/pip --version
pip 22.2.2 from /home/clark/tmp/venv1/lib64/python3.10/site-packages/pip (python 3.10)
> venv1/bin/pip install build wheel
Collecting build
  Using cached build-0.8.0-py3-none-any.whl (17 kB) 
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) 
Collecting packaging>=19.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB) 
Collecting pep517>=0.9.1
  Using cached pep517-0.13.0-py3-none-any.whl (18 kB) 
Collecting tomli>=1.0.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB) 
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) 
Installing collected packages: wheel, tomli, pyparsing, pep517, packaging, build
Successfully installed build-0.8.0 packaging-21.3 pep517-0.13.0 pyparsing-3.0.9 tomli-2.0.1 wheel-0.37.1

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip 
> venv1/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Downloading netifaces-0.11.0.tar.gz (30 kB) 
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: netifaces
  Building wheel for netifaces (setup.py) ... done
  Created wheel for netifaces: filename=netifaces-0.11.0-cp310-cp310-linux_x86_64.whl size=35306 sha256=0bc5bfaf024c86104588db92a0124cf6d181d45c1935bccd75ec0812b7327644
  Stored in directory: /home/clark/tmp/pip-testing/wheels/wheels/48/65/b3/4c4cc6038b81ff21cc9df69f2b6774f5f52e23d3c275ed15aa
Successfully built netifaces
Installing collected packages: netifaces
Successfully installed netifaces-0.11.0

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip 
> python3 -m venv venv2
> venv2/bin/python --version
Python 3.10.7
> venv2/bin/pip --version
pip 22.2.2 from /home/clark/tmp/venv2/lib64/python3.10/site-packages/pip (python 3.10)
> venv2/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Using cached netifaces-0.11.0-cp310-cp310-linux_x86_64.whl
Installing collected packages: netifaces
Successfully installed netifaces-0.11.0

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip

And now using the same wheel cache but with a different virtualenv that has newer pip installed it fallsback to building from the sdist:

> python3 -m venv venv2
> venv2/bin/python --version
Python 3.10.7
> venv2/bin/pip --version
pip 22.3 from /home/clark/tmp/venv2/lib64/python3.10/site-packages/pip (python 3.10)
> venv2/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Using cached netifaces-0.11.0.tar.gz (30 kB)
  Preparing metadata (setup.py) ... done
Installing collected packages: netifaces
  DEPRECATION: netifaces is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for netifaces ... done
Successfully installed netifaces-0.11.0

Note netifaces was chosen as an example here because it does not publish python3.10 wheels on pypi and its install is quick and simple. I don’t expect this to be a package specific issue, but I suppose that is possible as well.

Output

Output was provided inline with the reproduction steps above.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[cboylan]

@cboylan - just a comment on rhe use case of yours. We are using a different (and IMHO better) approach in multi-staging image of Airflow. Rather than relying on building and copying either wheels or cache between the stages, we are using --user§ flag in the build stage and then we copy the whole resulting .local directory between stages.

This has the advantage that it is guaranteed to work and it has smallest possible overhead for copying the files - it only copies what is needed and size of the final image is as small as possible.

Yup, I know that was some of the intent around the system we currently have. I’m not sure why it was built to copy wheels across and do another install there since as you mention that increases the image by the size of the wheels themselves. I’ll have to investigate if we can shift to this method of using user installed packages/a virtualenv to reduce the image sizes further. At the very least, I think child images will need to update their ENTRYPOIN/CMD directives. Thank you for calling this out, I suspect that this would be an improvement for us.

[pradyunsg]

FWIW, I’ll note that in #11143 – the fact that it requires changing the cache format and the lack of a migration path for the same, was treated as a blocking concern.

if cache layout/format changes for pip are something which should be mentioned in release notes

I don’t think we intentionally changed the cache format here. It’s probably reasonable to treat this as a bug/regression IMO. 😃