pip>=20.2 needlessly re-installs dependencies available on the system in system-site-packages venv

What did you want to do?

We are using a venv that explicitly does use system-site-packages, in order to allow users in a container, which does not allow write access to /, to install their own packages while still using all the (partially very complex to install/build) packages provided by the container image. This used to work fine until pip>=20.2, and is still broken in latest pip dev build as of today.

Output

Singularity> python3 -m venv --system-site-packages test_venv
Singularity> source ./test_venv/bin/activate
(test_venv) Singularity> pip list | grep numpy
numpy              1.19.1
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/home/users/fmaussion/test_venv/bin/python3 -m pip install --upgrade pip' command.
(test_venv) Singularity> pip install numpy
Requirement already satisfied: numpy in /usr/local/pyenv/versions/3.7.8/lib/python3.7/site-packages (1.19.1)
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/home/users/fmaussion/test_venv/bin/python3 -m pip install --upgrade pip' command.
(test_venv) Singularity> pip install --upgrade pip setuptools
Collecting pip
  Using cached pip-20.2.1-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
  Using cached setuptools-49.2.1-py3-none-any.whl (789 kB)
Installing collected packages: pip, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
  Attempting uninstall: setuptools
    Found existing installation: setuptools 47.1.0
    Uninstalling setuptools-47.1.0:
      Successfully uninstalled setuptools-47.1.0
Successfully installed pip-20.2.1 setuptools-49.2.1
(test_venv) Singularity> pip install numpy
Collecting numpy
  Using cached numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy
Successfully installed numpy-1.19.1

As you can see, before updating pip to 20.2, it correctly notices that numpy is already installed and does not need to do anything. After the update, it goes and installs a copy of the same numpy version in the venv. The same happens when installing a package that depends on numpy. Also the same goes for any other dependency that’s already provided by the system site-packages. “pip list” correctly lists numpy as installed in either case.

Additional information

The reason we are doing it this way is because building some of those dependencies, which are binary and annoying to deal with (most prominently gdal and fiona). The singularity container image has all of them pre-built, with optimizations specific for the host systems CPU.

For now I’ve instructed users to either use --no-deps when installing things, or to explicitly pin pip to <20.2.

[sbidoul]

@fdebrabander the issue is fixed in #8702 and should be part of the next patch release. You can test it by installing that branch with pip install -U "pip @ git+https://github.com/pypa/pip@refs/pull/8702/head".

[sbidoul]

This is due to the local_only argument of search_distributions that is True by default when called from utils.misc.get_distribution().