2020 resolver does not consider hashes in the constraints file

See:

• https://github.com/jwhitlock/pip-resolver-demo for a standalone demo of this issue
• https://github.com/jwhitlock/pip-resolver-demo/tree/original for the original, more complex standalone demo of this issue
• https://github.com/mozilla/ichnaea/pull/1289 for the full project

Update 1: The issue was not the projects multiple requirements file, but instead the combinations of using hashes and a constraints files. The current resolver uses hashes on a requirement in a constraints file, while the 2020 resolver ignores them, and fails to install because they do not have hashes. The description below describes the more complex version.

Update 2: Merged in the changes, so the default branch of pip-resolver-demo describes the simpler version, and includes the django.txt / django-versions.txt example as well.

What did you want to do?

Our requirements files include other files, as a way to only specify a requirement once for two different environments (development and building in ReadTheDocs.org):

• default.txt: -c constraints.txt, -r docs.txt, -r shared.txt
• docs.txt: -c constraints.txt, -r shared.txt
• shared.txt: None
• constraints.txt: None

All of our requirements are specified with hashes, populated with hashin.

This works, without warnings, when installing with pip install -r default.txt and pip 20.2.2.

When installing with pip 20.2.2 or pip-20.3.dev0 (today’s in-development version), this fails:

pip install -r default.txt --use-feature=2020-resolver
....
Collecting idna<3,>=2.5
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r shared.txt (line 6))

The idna==2.10 requirement is in constraints.txt, with the sha256 hash.

A similar error occurs when installing docs.txt:

pip install -r docs.txt --use-feature=2020-resolver
...
Collecting chardet<4,>=3.0.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    chardet<4,>=3.0.2 from https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl#sha256=fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 (from requests[security]==2.24.0->-r shared.txt (line 6))

Output

Here’s pip install -r default.txt --use-feature=2020-resolver:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting geoip2==4.0.2
  Downloading geoip2-4.0.2-py2.py3-none-any.whl (25 kB)
Collecting maxminddb==2.0.2
  Downloading maxminddb-2.0.2.tar.gz (285 kB)
Collecting idna<3,>=2.5
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r shared.txt (line 6))

Here’s pip install -r docs.txt --use-feature=2020-resolver:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting chardet<4,>=3.0.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    chardet<4,>=3.0.2 from https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl#sha256=fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 (from requests[security]==2.24.0->-r shared.txt (line 6))

For comparison, here’s pip install -r default.txt:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Collecting geoip2==4.0.2
  Downloading geoip2-4.0.2-py2.py3-none-any.whl (25 kB)
Collecting maxminddb==2.0.2
  Downloading maxminddb-2.0.2.tar.gz (285 kB)
Collecting idna==2.10
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet==3.0.4
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting pyOpenSSL==19.1.0
  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting cryptography==3.0
  Downloading cryptography-3.0-cp35-abi3-manylinux2010_x86_64.whl (2.7 MB)
Collecting packaging==20.4
  Downloading packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting Babel==2.8.0
  Downloading Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)
Collecting docutils==0.15.2
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting sphinxcontrib-applehelp==1.0.2
  Downloading sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting imagesize==1.2.0
  Downloading imagesize-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting Jinja2==2.11.2
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting sphinxcontrib-devhelp==1.0.2
  Downloading sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Collecting sphinxcontrib-htmlhelp==1.0.3
  Downloading sphinxcontrib_htmlhelp-1.0.3-py2.py3-none-any.whl (96 kB)
Collecting alabaster==0.7.12
  Downloading alabaster-0.7.12-py2.py3-none-any.whl (14 kB)
Collecting sphinxcontrib-jsmath==1.0.1
  Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Collecting Pygments==2.6.1
  Downloading Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting snowballstemmer==2.0.0
  Downloading snowballstemmer-2.0.0-py2.py3-none-any.whl (97 kB)
Collecting sphinxcontrib-serializinghtml==1.1.4
  Downloading sphinxcontrib_serializinghtml-1.1.4-py2.py3-none-any.whl (89 kB)
Collecting sphinxcontrib-qthelp==1.0.3
  Downloading sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Collecting aiohttp==3.6.2
  Downloading aiohttp-3.6.2-py3-none-any.whl (441 kB)
Collecting six==1.15.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting cffi==1.14.1
  Downloading cffi-1.14.1-cp38-cp38-manylinux1_x86_64.whl (409 kB)
Collecting pyparsing==2.4.7
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pytz==2020.1
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting MarkupSafe==1.1.1
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Collecting multidict==4.7.6
  Downloading multidict-4.7.6-cp38-cp38-manylinux1_x86_64.whl (162 kB)
Collecting yarl==1.5.1
  Downloading yarl-1.5.1-cp38-cp38-manylinux1_x86_64.whl (262 kB)
Collecting async-timeout==3.0.1
  Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting attrs==19.3.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting pycparser==2.20
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: MarkupSafe, maxminddb
  Building wheel for MarkupSafe (setup.py): started
  Building wheel for MarkupSafe (setup.py): finished with status 'done'
  Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-py3-none-any.whl size=12629 sha256=2f6a74f92d30e9e5c58a60f6b78a85229e279ee2229e4fced3d0c2c4e25c3d11
  Stored in directory: /root/.cache/pip/wheels/0c/61/d6/4db4f4c28254856e82305fdb1f752ed7f8482e54c384d8cb0e
  Building wheel for maxminddb (setup.py): started
  Building wheel for maxminddb (setup.py): finished with status 'done'
  Created wheel for maxminddb: filename=maxminddb-2.0.2-py3-none-any.whl size=15259 sha256=015ad8503c939cf424e3fedf475b0a3e81dd4dffbc95ed04e55235b8ce8d9f3c
  Stored in directory: /root/.cache/pip/wheels/8f/e8/68/7354267262db5ca21a7f869d544e703dcd7227b244db984b52
Successfully built MarkupSafe maxminddb
Installing collected packages: alabaster, pytz, Babel, pycparser, cffi, chardet, six, cryptography, docutils, idna, imagesize, MarkupSafe, Jinja2, Pygments, pyOpenSSL, snowballstemmer, sphinxcontrib-applehelp, sphinxcontrib-htmlhelp, sphinxcontrib-jsmath, sphinxcontrib-devhelp, sphinxcontrib-serializinghtml, pyparsing, packaging, sphinxcontrib-qthelp, attrs, multidict, yarl, async-timeout, aiohttp, certifi, urllib3, requests, Sphinx, maxminddb, geoip2
Successfully installed Babel-2.8.0 Jinja2-2.11.2 MarkupSafe-1.1.1 Pygments-2.6.1 Sphinx-3.1.2 aiohttp-3.6.2 alabaster-0.7.12 async-timeout-3.0.1 attrs-19.3.0 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 cryptography-3.0 docutils-0.15.2 geoip2-4.0.2 idna-2.10 imagesize-1.2.0 maxminddb-2.0.2 multidict-4.7.6 packaging-20.4 pyOpenSSL-19.1.0 pycparser-2.20 pyparsing-2.4.7 pytz-2020.1 requests-2.24.0 six-1.15.0 snowballstemmer-2.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 urllib3-1.25.10 yarl-1.5.1

Additional information

Full dependency tree:

geoip2==4.0.2
  - aiohttp [required: >=3.6.2,<4.0.0, installed: 3.6.2]
    - async-timeout [required: >=3.0,<4.0, installed: 3.0.1]
    - attrs [required: >=17.3.0, installed: 19.3.0]
    - chardet [required: >=2.0,<4.0, installed: 3.0.4]
    - multidict [required: >=4.5,<5.0, installed: 4.7.6]
    - yarl [required: >=1.0,<2.0, installed: 1.5.1]
      - idna [required: >=2.0, installed: 2.10]
      - multidict [required: >=4.0, installed: 4.7.6]
  - maxminddb [required: >=2.0.0,<3.0.0, installed: 2.0.2]
  - requests [required: >=2.24.0,<3.0.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - urllib3 [required: >=1.25.2,<2.0.0, installed: 1.25.10]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.2.2]
pyOpenSSL==19.1.0
  - cryptography [required: >=2.8, installed: 3.0]
    - cffi [required: >=1.8,!=1.11.3, installed: 1.14.1]
      - pycparser [required: Any, installed: 2.20]
    - six [required: >=1.4.1, installed: 1.15.0]
  - six [required: >=1.5.2, installed: 1.15.0]
Sphinx==3.1.2
  - alabaster [required: >=0.7,<0.8, installed: 0.7.12]
  - babel [required: >=1.3, installed: 2.8.0]
    - pytz [required: >=2015.7, installed: 2020.1]
  - docutils [required: >=0.12, installed: 0.15.2]
  - imagesize [required: Any, installed: 1.2.0]
  - Jinja2 [required: >=2.3, installed: 2.11.2]
    - MarkupSafe [required: >=0.23, installed: 1.1.1]
  - packaging [required: Any, installed: 20.4]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - six [required: Any, installed: 1.15.0]
  - Pygments [required: >=2.0, installed: 2.6.1]
  - requests [required: >=2.5.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - setuptools [required: Any, installed: 49.6.0]
  - snowballstemmer [required: >=1.1, installed: 2.0.0]
  - sphinxcontrib-applehelp [required: Any, installed: 1.0.2]
  - sphinxcontrib-devhelp [required: Any, installed: 1.0.2]
  - sphinxcontrib-htmlhelp [required: Any, installed: 1.0.3]
  - sphinxcontrib-jsmath [required: Any, installed: 1.0.1]
  - sphinxcontrib-qthelp [required: Any, installed: 1.0.3]
  - sphinxcontrib-serializinghtml [required: Any, installed: 1.1.4]
wheel==0.35.1

[uranusjr]

@jwhitlock I think the fix to the issue is surfacing another different bug (that was not triggered previously since pip fails before hitting it). It would be best to track it in another issue.

[uranusjr]

The merging is done in the Hashes class (which is on InstallRequirement). I traced the code and believe it’s possible to only change the new resolver implementation to do the intersection logic (and keep the legacy resolver as-is, performing union). PR coming shortly.