pip resolver installing incompatible versions

Description

See reproduce steps for setup.

It installed certifi==2021.5.30, and then reported an error for a version mismatch. Instead, pip should have looked at the requirements and installed 2020.12.5 to work with all the other dependencies.

It seems that passing --upgrade makes it work.

Ref: https://github.com/linkedin/datahub/issues/2717

Expected behavior

It should have installed certifi==2020.12.5, since the main requirements are certifi<2021.0.0 and greater than a version from 2017.

pip version

21.1.2

Python version

3.8.6

OS

MacOS

How to Reproduce

The following steps:

python3.8 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools  # sanity check
pip freeze  # empty, since I'm in a fresh venv
PIP_RESOLVER_DEBUG=1 pip install 'acryl-datahub[snowflake]==0.8.1.2'

Yields:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
snowflake-connector-python 2.4.5 requires certifi<2021.0.0, but you have certifi 2021.5.30 which is incompatible.

Output

Edited by @uranusjr: I put this into a Gist instead since the log is way too long to reasonably read and reference in comments.

https://gist.github.com/uranusjr/74ec344ce72f9c9cbe91b262bcd49dd5

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

I dug into for like two hours and finally think I know what’s going on.

The trigger to this issue is that snowflake-connector-python specifies certifi twice. From the wheel’s metadata:

Metadata-Version: 2.1
Name: snowflake-connector-python
Version: 2.4.5
...
Requires-Dist: certifi (<2021.0.0)
...
Requires-Dist: certifi (>=2017.4.17)
...

A wrikle to this is that there’s a reproducibility issue, because the issue only happen when certifi<2021.0.0 is fed into the resolver before certifi>=2017.4.17. But pkg_resources shuffles the Requires-Dist values on calculation, so to reproduce this deterministically, you need to change line 3041 of pkg_resources.DistInfoDistribution._compute_dependencies() to something like:

dm[None].extend(reqs_for_extra(None))

so the order of requirements don’t change between runs.

Now onto the actual resolver stuff. When the resolver pins through a candidate, it converts all of the candidate’s dependencies into criteria so they can be merged into existing criteria:

https://github.com/pypa/pip/blob/7c3abccd22c4bbd6c1da2ae41845ffbd26c1bb17/src/pip/_vendor/resolvelib/resolvers.py#L200-L205

But it uses a dict for the job. So when a candidate specifies a package multiple times (like snowflake-connector-python), the latter specification would override the previous one. Since certifi<2021.0.0 is specified first (after we patch pkg_resources; without the patch the resolver has 50% chance to get the “right” one), it got lost during resolution.

I have to say, I haven’t had such an exercise debugging an issue like this for quite a while. This kind of things is why I’m doing open source work, so thanks for that, in a way 😆 I’m unfortunately quite busy tomorrow (it’s late Saturday here), but will try to find some time to fix asap.

[uranusjr]

I can reproduce this in a fresh virtual environment. This is very weird. I suspect there is a package doing funky things to bypass the resolver and install its own things.