Dependency resolver downloads each version of one of two requested packages, due to choosing version of a dependency before inspecting both requested packages.

What did you want to do?

In a fresh virtual environment, I wish to install two locally developed and hosted packages, jbafarm (v1.10.31) and litmus (v1.4.2). These packages have a single common dependency, gdal, which we also have internally hosted versions of. The latest version of jbafarm has the requirement gdal==2.2.0, while litmus has gdal>=2.2.0. gdal 2.2.0 is available on the package server, and is clearly (to the human observer) the only obvious resolution.

Output

When one tries install a specific version of jbafarm, the install proceeds successfully - gdal==2.2.0 is picked up, satisfying both packages’ requirements:

$ pip install "jbafarm==1.10.31" litmus --use-feature=2020-resolver        
Looking in indexes: https://pypi.org/simple, http://<internal url>
Collecting jbafarm==1.10.31
  Downloading http://<internal url>/packages/jbafarm-1.10.31-cp36-cp36m-linux_x86_64.whl (822 kB)
     |████████████████████████████████| 822 kB 33.7 MB/s
Collecting gdal==2.2.0
  Downloading http://<internal url>/packages/GDAL-2.2.0-cp36-cp36m-manylinux1_x86_64.whl (23.7 MB)
     |████████████████████████████████| 23.7 MB 47.3 MB/s
Collecting litmus
  Downloading http://<internal url>/packages/litmus-1.4.2-py2.py3-none-any.whl (18 kB)

...

However, when the jbafarm requirement is left open, the resolver ‘fails’ by downloading each version of jbafarm individually. We’ve not yet seen what happens when it finally exhausts all versions - timeouts on CI builds, on the package server, or our own patience kick in!

The issue appears to be that Pip downloads litmus, picks up the gdal>=2.2.0 requirement, and settles on the latest available version (gdal==3.1.3). Only after this does it download the latest jbafarm, and finds this does not resolve that GDAL requirement. It then downloads the previous version of jbafarm, finds the same, and so on.

$ pip install jbafarm litmus --use-feature=2020-resolver
Looking in indexes: https://pypi.org/simple, http://<internal url>
Collecting litmus
  Downloading http://<internal url>/packages/litmus-1.4.2-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: wheel in /home/jbanorthwest.co.uk/danielevans/venvs/farmmap3/lib/python3.6/site-packages (from litmus) (0.35.1)
Collecting gdal>=2.2.0
  Downloading http://<internal url>/packages/GDAL-3.1.3-cp36-cp36m-manylinux1_x86_64.whl (28.4 MB)
     |████████████████████████████████| 28.4 MB 35.5 MB/s
Collecting jbafarm
  Downloading http://<internal url>/packages/jbafarm-1.10.31-cp36-cp36m-linux_x86_64.whl (822 kB)
     |████████████████████████████████| 822 kB 41.4 MB/s
  Downloading http://<internal url>/packages/jbafarm-1.10.28-cp36-cp36m-linux_x86_64.whl (821 kB)
     |████████████████████████████████| 821 kB 39.3 MB/s
  Downloading http://<internal url>/packages/jbafarm-1.10.27-cp36-cp36m-linux_x86_64.whl (816 kB)
     |████████████████████████████████| 816 kB 28.3 MB/s
  Downloading http://<internal url>/packages/jbafarm-1.10.26-cp36-cp36m-linux_x86_64.whl (816 kB)
     |████████████████████████████████| 816 kB 29.7 MB/s
  Downloading http://<internal url>/packages/jbafarm-1.10.25-cp36-cp36m-linux_x86_64.whl (816 kB)
     |████████████████████████████████| 816 kB 38.8 MB/s
  Downloading http://<internal url>/packages/jbafarm-1.10.24-cp36-cp36m-linux_x86_64.whl (816 kB)
     |████████████████████████████████| 816 kB 27.5 MB/s
  Downloading http://<internal url>/packages/jbafarm-1.10.23-cp36-cp36m-linux_x86_64.whl (816 kB)
     |████████████████████████████████| 816 kB 37.7 MB/s

[Continues until timeout or cancelled]

Additional information

We use an instance of pypi-server (version 1.3.2) to host these packages, and configure Pip to use it via --extra-index-url (either CLI or via Pip’s config file).

I’m surprised that Pip resorts to downloading each jbafarm version in turn. I’ve never seen this behaviour before - does Pip usually use some metadata from PyPI to avoid all these downloads, which our local pypi-server instance is missing? Either way, I’m surprised that it doesn’t check the requirements for (the latest versions of) all user-specified packages before settling on some resolutions.

It may be that Pip changes the gdal choice after inspecting every jbafarm version, but it’ll take a very long time to achieve that. It doesn’t seem like a good tactic is to search older versions of a package to see if it ever allowed a higher version of some dependency; firstly, because it seems fairly unlikely that an older version of a package allowed a higher version of a dependency, and secondly, because as a user I expect it to install the latest possible version of the specified package, even if some dependencies aren’t the latest available, rather than vice versa.

The issue still occurs on the current development version of Pip. Please say if you’d like some more verbose logging output - I didn’t include it initially as pip -v is very verbose.

[uranusjr]

@DanielFEvans I made some changes to the resolver logic. Can you modify your pip insallation locally and check whether the resolver can catch the conflict earlier?

Code is at https://github.com/pypa/pip/pull/8924/files

[brainwane]

Now that #8924 is merged I’m marking this as closed, since per https://github.com/pypa/pip/issues/8893#issuecomment-699842201 it looks like the original issue is resolved and we already know about the visual quirk. Thanks, everyone!