[2020-resolver & fast-deps] BadZipFile when force-reinstalling packages

Environment

• pip version: 20.3.dev0 at commit 5a61475f
• Python version: 3.7.7
• OS: Fedora 30

Description Force-reinstalling a cached package causes pip to fail if use-feature = 2020-resolver fast-deps both are enabled.

I saw this behavior with Black, pip itself, wheel and setuptools. The example below is for Black.

If only either 2020-resolver or fast-deps is enabled, everything works correctly.

Expected behavior The package should be reinstalled successfully.

How to Reproduce

1. Clear the pip cache
2. Create a virtualenv
3. Update pip to current master
4. Enable 2020-resolver and fast-deps using pip config set --site global.use-feature "2020-resolver fast-deps"
5. Install Black with pip
6. Install Black again using the --force-reinstall option

Output

$ rm ~/.cache/pip -rf
$ mktmpenv
Using base prefix '/usr'
New python executable in /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/bin/python3
Also creating executable in /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/bin/preactivate
virtualenvwrapper.user_scripts creating /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/bin/postactivate
virtualenvwrapper.user_scripts creating /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/bin/get_env_details
This is a temporary environment. It will be deleted when you run 'deactivate'.
$ python -m pip install -U "pip @ https://github.com/pypa/pip/archive/master.zip"
Collecting pip@ https://github.com/pypa/pip/archive/master.zip
  Using cached https://github.com/pypa/pip/archive/master.zip
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517) ... done
  Created wheel for pip: filename=pip-20.3.dev0-py2.py3-none-any.whl size=1503406 sha256=adbf64c5ebbd34c17295a6c96c5e90e5ca9f35714f75001e1c24c971e9232302
  Stored in directory: /tmp/pip-ephem-wheel-cache-b_6p5pik/wheels/cb/71/7f/677ff1340ac636bc57b0dda6815dade9ad667c5c337aa5dc39
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.1
    Uninstalling pip-20.2.1:
      Successfully uninstalled pip-20.2.1
Successfully installed pip-20.3.dev0
$ pip config set --site global.use-feature "2020-resolver fast-deps"                            
Writing to /home/akaihola/.virtualenvs/tmp-32e020b1a10180f/pip.conf
$ pip install black
WARNING: pip is using lazily downloaded wheels using HTTP range requests to obtain dependency information. This experimental feature is enabled through --use-feature=fast-deps and it is not ready for production.
Collecting black
  Obtaining dependency information from black 19.10b0
Collecting typed-ast>=1.4.0
  Obtaining dependency information from typed-ast 1.4.1
Collecting pathspec<1,>=0.6
  Obtaining dependency information from pathspec 0.8.0
Collecting toml>=0.9.4
  Obtaining dependency information from toml 0.10.1
Collecting attrs>=18.1.0
  Obtaining dependency information from attrs 19.3.0
Collecting click>=6.5
  Obtaining dependency information from click 7.1.2
Collecting appdirs
  Obtaining dependency information from appdirs 1.4.4
Collecting regex
  Obtaining dependency information from regex 2020.7.14
Collecting black
  Downloading black-19.10b0-py36-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 1.2 MB/s
Collecting typed-ast>=1.4.0
  Downloading typed_ast-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (737 kB)
     |████████████████████████████████| 737 kB 2.5 MB/s
Collecting pathspec<1,>=0.6
  Downloading pathspec-0.8.0-py2.py3-none-any.whl (28 kB)
Collecting toml>=0.9.4
  Downloading toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting attrs>=18.1.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting click>=6.5
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 1.1 MB/s
Collecting appdirs
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting regex
  Downloading regex-2020.7.14-cp37-cp37m-manylinux2010_x86_64.whl (660 kB)
     |████████████████████████████████| 660 kB 3.8 MB/s
Installing collected packages: typed-ast, toml, regex, pathspec, click, attrs, appdirs, black
Successfully installed appdirs-1.4.4 attrs-19.3.0 black-19.10b0 click-7.1.2 pathspec-0.8.0 regex-2020.7.14 toml-0.10.1 typed-ast-1.4.1
$ pip install --force-reinstall black
WARNING: pip is using lazily downloaded wheels using HTTP range requests to obtain dependency information. This experimental feature is enabled through --use-feature=fast-deps and it is not ready for production.
Collecting black
  Obtaining dependency information from black 19.10b0
ERROR: black has an invalid wheel, could not read 'black-19.10b0.dist-info/LICENSE' file: BadZipFile('Bad magic number for file header')

The final error for the other packages I tried was:

# wheel (no traceback):
ERROR: wheel has an invalid wheel, could not read 'wheel-0.34.2.dist-info/LICENSE.txt' file: BadZipFile('Bad magic number for file header')

# setuptools and pip:
  File "/home/kaiant/.virtualenvs/tmp-c9ddecd5fb033a7/lib/python3.7/site-packages/pip/_internal/network/lazy_wheel.py", line 46, in dist_from_wheel_url
    zip_file = ZipFile(wheel)  # type: ignore
  File "/usr/lib64/python3.7/zipfile.py", line 1258, in __init__
    self._RealGetContents()
  File "/usr/lib64/python3.7/zipfile.py", line 1353, in _RealGetContents
    raise BadZipFile("Bad magic number for central directory")
zipfile.BadZipFile: Bad magic number for central directory

[McSinyx]

The problem seem to be with incorrect (?) caching of range responses (please have my apologies, @akaihola, for putting you through creating a container for this—I have --no-cache-dir habitually added to every command testing the fast-deps feature and thus didn’t catch the error). The following patch would hot fix the problem and leave caching to be revisit later:

diff --git a/src/pip/_internal/network/lazy_wheel.py b/src/pip/_internal/network/lazy_wheel.py
index 16be0d29..c03eaeb8 100644
--- a/src/pip/_internal/network/lazy_wheel.py
+++ b/src/pip/_internal/network/lazy_wheel.py
@@ -194,7 +194,8 @@ class LazyZipOverHTTP(object):
     def _stream_response(self, start, end, base_headers=HEADERS):
         # type: (int, int, Dict[str, str]) -> Response
         """Return HTTP response to a range request from start to end."""
-        headers = {'Range': 'bytes={}-{}'.format(start, end)}
+        headers = {'Range': 'bytes={}-{}'.format(start, end),
+                   'Cache-Control': 'no-cache'}
         headers.update(base_headers)
         return self._session.get(self._url, headers=headers, stream=True)

[McSinyx]

Thank you @akaihola, this is really helpful, I’ll investigate this ASAP. Until then, I want to note that the fast-deps feature is no-op when used with the legacy resolver (i.e. when 2020-resolver is not enabled) and that shallow wheels are not cached (I’m not sure if it’s a good idea, I’ve never really thought about it).