Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

cache doesn't seem to work

See original GitHub issue

Pip version: 9.0.1
Python version: 2.7.12
Operating system: Ubuntu 17.04

Description:

When using a custom Artifactory for PyPi the cache doesn’t seem to work, it gets downloaded again, even though there is an age in the cache:

(that) zoro@uvm ~/work/git/magic (master●●)$ /home/zoro/work/git/magic/.tox/pip/bin/pip install --disable-pip-version-check -vvv --cache-dir /home/zoro/temp/pip -i https://magic.com/artifactory/api/pypi/pypi-repos/simple "pluggy"
Collecting pluggy
  1 location(s) to search for versions of pluggy:
  * https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/
  Getting page https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/
  Looking up "https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): magic.com
  "GET /artifactory/api/pypi/pypi-repos/simple/pluggy/ HTTP/1.1" 200 None
  Updating cache with response from "https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/"
  Analyzing links from page https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/7a/09/56dd120ac549f335cc43e777432c5cca15c3c9cfeae8685ea5bd1a5e3779/pluggy-0.3.0-py2.py3-none-any.whl#md5=575b0796bc8faf39e5cd65a07436db20 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.0
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/c3/61/1d23c57fcc6a5dcdf4ffd25ea1d567a8462762fab74ab9bb20df62b4a063/pluggy-0.3.0.tar.gz#md5=7e9e718884cac045efe1400c7ad016b5 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.0
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/d3/ff/d63342af76d20f9b94c2138aef8739df23311e3ce8a8ae7e4b0733ab0827/pluggy-0.3.1-py2.py3-none-any.whl#md5=bc68660e5e5cf7242ccdc0725de84c3c (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.1
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/1b/a9/6f5f80b75a8d84d21a8a13486fe26a2da9f043f93b464b2e3928be256dc4/pluggy-0.3.1.tar.gz#md5=ecdd791e309f60668b66fec97c2ee7db (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.1
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl#md5=df144953fde470f504573b32655259e2 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.4.0
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/38/e2/b23434f4030bbb1af3bcdbb2ecff6b11cf2e467622446ce66a08e99f2ea9/pluggy-0.4.0.zip#md5=447a92368175965d2fbacaef9f3df842 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.4.0
  Using version 0.4.0 (newest of versions: 0.3.0, 0.3.1, 0.4.0)
  Looking up "https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl" in the cache
  No cache entry available
  Resetting dropped connection: magic.com
  Starting new HTTPS connection (2): magic.com
  "GET /artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl HTTP/1.1" 200 11900
  Downloading https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl
  Downloading from URL https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl#md5=df144953fde470f504573b32655259e2 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/)
  Updating cache with response from "https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl"
  Caching due to etag
Installing collected packages: pluggy

Successfully installed pluggy-0.4.0
Cleaning up...


(that) zoro@uvm ~/work/git/magic (master●●)$ rm -rf ./.tox/pip/lib/python2.7/site-packages/pluggy*


(that) zoro@uvm ~/work/git/magic (master●●)$ /home/zoro/work/git/magic/.tox/pip/bin/pip install --disable-pip-version-check -vvv --cache-dir /home/zoro/temp/pip -i https://magic.com/artifactory/api/pypi/pypi-repos/simple "pluggy"
Collecting pluggy
  1 location(s) to search for versions of pluggy:
  * https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/
  Getting page https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/
  Looking up "https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): magic.com
  "GET /artifactory/api/pypi/pypi-repos/simple/pluggy/ HTTP/1.1" 200 None
  Updating cache with response from "https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/"
  Analyzing links from page https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/7a/09/56dd120ac549f335cc43e777432c5cca15c3c9cfeae8685ea5bd1a5e3779/pluggy-0.3.0-py2.py3-none-any.whl#md5=575b0796bc8faf39e5cd65a07436db20 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.0
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/c3/61/1d23c57fcc6a5dcdf4ffd25ea1d567a8462762fab74ab9bb20df62b4a063/pluggy-0.3.0.tar.gz#md5=7e9e718884cac045efe1400c7ad016b5 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.0
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/d3/ff/d63342af76d20f9b94c2138aef8739df23311e3ce8a8ae7e4b0733ab0827/pluggy-0.3.1-py2.py3-none-any.whl#md5=bc68660e5e5cf7242ccdc0725de84c3c (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.1
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/1b/a9/6f5f80b75a8d84d21a8a13486fe26a2da9f043f93b464b2e3928be256dc4/pluggy-0.3.1.tar.gz#md5=ecdd791e309f60668b66fec97c2ee7db (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.3.1
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl#md5=df144953fde470f504573b32655259e2 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.4.0
    Found link https://magic.com/artifactory/api/pypi/pypi-repos/packages/38/e2/b23434f4030bbb1af3bcdbb2ecff6b11cf2e467622446ce66a08e99f2ea9/pluggy-0.4.0.zip#md5=447a92368175965d2fbacaef9f3df842 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/), version: 0.4.0
  Using version 0.4.0 (newest of versions: 0.3.0, 0.3.1, 0.4.0)
  Looking up "https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl" in the cache
  Current age based on date: 13
  Resetting dropped connection: magic.com
  Starting new HTTPS connection (2): magic.com
  "GET /artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl HTTP/1.1" 200 11900
  Downloading https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl
  Downloading from URL https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl#md5=df144953fde470f504573b32655259e2 (from https://magic.com/artifactory/api/pypi/pypi-repos/simple/pluggy/)
  Updating cache with response from "https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl"
  Caching due to etag
Installing collected packages: pluggy

Successfully installed pluggy-0.4.0
Cleaning up...

Anyone can help me to find out what’s the problem? Thanks!

Issue Analytics

State:
Created 6 years ago
Comments:7 (4 by maintainers)

Top GitHub Comments

1reaction

dstufftcommented, Apr 3, 2017

Ok so the problem here is that Artifactory isn’t returning any Cache-Control headers, so we can’t cache the response and serve it without hitting the network. On top of that, while they do serve ETag headers on the file URLs, (which means we will store that response for later), when we attempt to do a conditional GET to that URL to tell us if we can reuse that stored response, they tell us that no, we can’t.

So essentially this is an Artifactory bug, they need to implement Cache-Control headers and conditional GET support.

0reactions

gaborbernatcommented, Apr 3, 2017

curl -x $http_proxy -I -H "If-None-Match: f346586bca1fd92c07dad2a9aa3768829b6e6816" https://magic.com/artifactory/api/pypi/pypi-repos/packages/52/5e/3e63895c9730f37195db7d3f3a6465b7ffce08871ca217947b81f452ce62/pluggy-0.4.0-py2.py3-none-any.whl
HTTP/1.1 200 Connection established

HTTP/1.1 200 OK
Server: Artifactory/4.14.3
X-Artifactory-Id: addad6265f72b35e618e07a44c16e48d34c388d4
X-Artifactory-Node-Id: asdas
Last-Modified: Sun, 25 Sep 2016 23:05:19 GMT
ETag: f346586bca1fd92c07dad2a9aa3768829b6e6816
X-Checksum-Sha1: f346586bca1fd92c07dad2a9aa3768829b6e6816
Accept-Ranges: bytes
X-Checksum-Md5: df144953fde470f504573b32655259e2
X-Artifactory-Filename: pluggy-0.4.0-py2.py3-none-any.whl
Content-Disposition: attachment; filename="pluggy-0.4.0-py2.py3-none-any.whl"; filename*=UTF-8''pluggy-0.4.0-py2.py3-none-any.whl
Content-Type: application/octet-stream
Content-Length: 11900
Date: Mon, 03 Apr 2017 13:33:19 GMT
Connection: close