Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Content is read from 304 response

See original GitHub issue

Environment

pip version: 19.2.3
Python version: 2.7, 3.6
OS: Linux

Pypi server: local Artifactory Highly concurrent tox/pip sessions.

Description pip receives a 304 response and consumes the response as if it were valid data.

The log shows that the 304 response with size 0 is received.

Created temporary directory: /tmp/pip-unpack-a3hgm2dy
Looking up "pypiserver.local/api/pypi/python/packages/packages/b8/12/d6e2a96db63c1bf43158e3ce4502c349dbcdcc338998976fedb402cce031/oslo.i18n-3.19.0-py2.py3-none-any.whl" in the cache
Current age based on date: 1
pypiserver.local:443 "GET /api/pypi/python/packages/packages/b8/12/d6e2a96db63c1bf43158e3ce4502c349dbcdcc338998976fedb402cce031/oslo.i18n-3.19.0-py2.py3-none-any.whl HTTP/1.1" 304 0
Downloading pypiserver.local/api/pypi/python/packages/packages/b8/12/d6e2a96db63c1bf43158e3ce4502c349dbcdcc338998976fedb402cce031/oslo.i18n-3.19.0-py2.py3-none-any.whl
Downloading from URL pypiserver.local/api/pypi/python/packages/packages/b8/12/d6e2a96db63c1bf43158e3ce4502c349dbcdcc338998976fedb402cce031/oslo.i18n-3.19.0-py2.py3-none-any.whl#sha256=7f58f6b5865313442948132361122b1972599ea4985af61e8d4b63e78093d5f8 (from pypiserver.local/api/pypi/python/simple/oslo-i18n/)

Then the execution goes on to download the content and the error is raised when the installation is attempted. The error is always something like:

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    oslo.i18n===3.19.0 from https://pypiserver.local/api/pypi/python/packages/packages/b8/12/d6e2a96db63c1bf43158e3ce4502c349dbcdcc338998976fedb402cce031/oslo.i18n-3.19.0-py2.py3-none-any.whl#sha256=7f58f6b5865313442948132361122b1972599ea4985af61e8d4b63e78093d5f8:
        Expected sha256 7f58f6b5865313442948132361122b1972599ea4985af61e8d4b63e78093d5f8
             Got        e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

The erroneous SHA value of e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 indicates NULL content.

The error is sporadic and random but due to the sheer number of pip sessions, the result is a very high error rate.

Expected behavior

Normally, the log messages should show that the cache was used:

  <server> "GET <uri> 304 0
  Using cached <url>
  Downloading from URL <url>#sha256=<sha>
  Added <package> from <url> to build tracker '<tracker>'
  Removed <package> from <url>#sha256=<sha> from build tracker '<tracker>'

In the erroneous case, the messages are:

<server> "GET <uri> 304 0
Downloading <url>
Downloading from URL <url>#sha256=<sha>

The expected behavior is that on 304 cache miss (that can obviously happen for some reason) the package is still downloaded and the cache is updated.

How to Reproduce

Given cachemiss_server.py

from http.server import BaseHTTPRequestHandler, HTTPServer


class CacheMissServer(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.requestline == 'GET /certifi/ HTTP/1.1':
            self.send_response(200, self.responses[200])
            self.send_header("Content-Type", "text/html; charset=UTF-8")
            self.end_headers()
            self.wfile.write(index_resp)
        else:
            self.send_response(304)
            self.end_headers()

index_resp = '''<html><head><title>Simple Index</title><meta name="api-version" value="2" /></head><body>
<a href="../../packages/packages/55/bf/e1094f052b5932a452b12ac5a6c229f3d7dbf41a66cfc8d5832fe3fb6463/certifi-0.0.1.tar.gz#sha256=a6f0d79b2b63a43c71bc128929be3a17bdf5d0a5f74208d7ad9563b3bf8210e9" rel="internal" >certifi-0.0.1.tar.gz</a><br/>
</body></html>
'''


if __name__ == '__main__':
    server = HTTPServer(('', 8888), CacheMissServer)
    server.serve_forever()

Run it with python2 cachemiss_server.py

Then run pip to see the error:

pip install -i http://localhost:8888 certifi -v

Created temporary directory: /tmp/pip-ephem-wheel-cache-jnh8j65n
Created temporary directory: /tmp/pip-req-tracker-7t1503qx
Created requirements tracker '/tmp/pip-req-tracker-7t1503qx'
Created temporary directory: /tmp/pip-install-z7wubh5n
Looking in indexes: http://localhost:8888
Collecting certifi
  1 location(s) to search for versions of certifi:
  * http://localhost:8888/certifi/
  Getting page http://localhost:8888/certifi/
  Found index url http://localhost:8888
  Starting new HTTP connection (1): localhost:8888
  http://localhost:8888 "GET /certifi/ HTTP/1.1" 200 None
  Analyzing links from page http://localhost:8888/certifi/
    Found link http://localhost:8888/packages/packages/55/bf/e1094f052b5932a452b12ac5a6c229f3d7dbf41a66cfc8d5832fe3fb6463/certifi-0.0.1.tar.gz#sha256=a6f0d79b2b63a43c71bc128929be3a17bdf5d0a5f74208d7ad9563b3bf8210e9 (from http://localhost:8888/certifi/), version: 0.0.1
  Given no hashes to check 1 links for project 'certifi': discarding no candidates
  Using version 0.0.1 (newest of versions: 0.0.1)
  Created temporary directory: /tmp/pip-unpack-c57gf32i
  Found index url http://localhost:8888
  Resetting dropped connection: localhost
  http://localhost:8888 "GET /packages/packages/55/bf/e1094f052b5932a452b12ac5a6c229f3d7dbf41a66cfc8d5832fe3fb6463/certifi-0.0.1.tar.gz HTTP/1.1" 304 0
  Downloading http://localhost:8888/packages/packages/55/bf/e1094f052b5932a452b12ac5a6c229f3d7dbf41a66cfc8d5832fe3fb6463/certifi-0.0.1.tar.gz
  Downloading from URL http://localhost:8888/packages/packages/55/bf/e1094f052b5932a452b12ac5a6c229f3d7dbf41a66cfc8d5832fe3fb6463/certifi-0.0.1.tar.gz#sha256=a6f0d79b2b63a43c71bc128929be3a17bdf5d0a5f74208d7ad9563b3bf8210e9 (from http://localhost:8888/certifi/)

Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-7t1503qx'
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    certifi from http://localhost:8888/packages/packages/55/bf/e1094f052b5932a452b12ac5a6c229f3d7dbf41a66cfc8d5832fe3fb6463/certifi-0.0.1.tar.gz#sha256=a6f0d79b2b63a43c71bc128929be3a17bdf5d0a5f74208d7ad9563b3bf8210e9:
        Expected sha256 a6f0d79b2b63a43c71bc128929be3a17bdf5d0a5f74208d7ad9563b3bf8210e9
             Got        e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Exception information:
Traceback (most recent call last):
  File "/home/akovi/openstack/mistral/.tox/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/home/akovi/openstack/mistral/.tox/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/home/akovi/openstack/mistral/.tox/venv/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 203, in resolve
    raise hash_errors
pip._internal.exceptions.HashErrors: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    certifi from http://localhost:8888/packages/packages/55/bf/e1094f052b5932a452b12ac5a6c229f3d7dbf41a66cfc8d5832fe3fb6463/certifi-0.0.1.tar.gz#sha256=a6f0d79b2b63a43c71bc128929be3a17bdf5d0a5f74208d7ad9563b3bf8210e9:
        Expected sha256 a6f0d79b2b63a43c71bc128929be3a17bdf5d0a5f74208d7ad9563b3bf8210e9
             Got        e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Issue Analytics

State:
Created 4 years ago
Comments:11 (6 by maintainers)

Top GitHub Comments

1reaction

andraskcommented, Sep 3, 2019

@chrahunt thanks for the note. I updated the description.

0reactions

andraskcommented, Oct 29, 2019

I found a job that still uses the ‘old mode’ with http cache. The last three builds from Oct 25 to Oct 28 have installed the 19.3.x version and the issue seems to not have occurred. I guess it endorses that the new cache solution probably works much better. I’ll close this issue. Great work! Woohoo 🚂