`content-type` for GCS wheel URL page not set consistently(?)

Hi JAX team. 👋 We (the pyhf team) have a slightly odd problem to report that came from https://github.com/scikit-hep/pyhf/pull/1579 following the removal of jaxlib v0.1.58 from PyPI for storage reasons (c.f. https://github.com/google/jax/discussions/7608#discussioncomment-1269382).

---

TL;DR: Is it possible that the content-type is not being set consistently for https://storage.googleapis.com/jax-releases/jax_releases.html across GCS servers somehow?

---

In our GitHub Actions based minimum supported dependencies workflow we’re installing the oldest release of jax and jaxlib with the oldest Python version that we claim pyhf supports, and to ensure that we have access to jaxlib wheels (once they have been removed from PyPI given the 10 GB ceiling) in PR https://github.com/scikit-hep/pyhf/pull/1579 we’re adding

--find-links https://storage.googleapis.com/jax-releases/jax_releases.html

to our lower-bound-requirements.txt.

Locally in a python:3.7 Docker container this works fine (for me)

$ docker run --rm -ti python:3.7 /bin/bash
root@f63cc4fbb2ff:/# python -m venv venv && . venv/bin/activate
(venv) root@f63cc4fbb2ff:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@f63cc4fbb2ff:/# pip --version
pip 21.2.4 from /venv/lib/python3.7/site-packages/pip (python 3.7)
(venv) root@f63cc4fbb2ff:/# python -m pip install "jaxlib==0.1.58" --find-links https://storage.googleapis.com/jax-releases/jax_releases.html
Looking in links: https://storage.googleapis.com/jax-releases/jax_releases.html
Collecting jaxlib==0.1.58
  Downloading https://storage.googleapis.com/jax-releases/cuda111/jaxlib-0.1.58%2Bcuda111-cp37-none-manylinux2010_x86_64.whl (182.1 MB)
     |████████████████████████████████| 182.1 MB 7.5 MB/s 
Collecting numpy>=1.12
  Downloading numpy-1.21.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
     |████████████████████████████████| 15.7 MB 5.9 MB/s 
Collecting absl-py
  Downloading absl_py-0.13.0-py3-none-any.whl (132 kB)
     |████████████████████████████████| 132 kB 40.8 MB/s 
Collecting flatbuffers
  Downloading flatbuffers-2.0-py2.py3-none-any.whl (26 kB)
Collecting scipy
  Downloading scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.5 MB)
     |████████████████████████████████| 28.5 MB 17.8 MB/s 
Collecting six
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, numpy, scipy, flatbuffers, absl-py, jaxlib
Successfully installed absl-py-0.13.0 flatbuffers-2.0 jaxlib-0.1.58+cuda111 numpy-1.21.2 scipy-1.7.1 six-1.16.0

However, in the GitHub Actions workflow this fails with

Successfully installed setuptools-57.4.0 wheel-0.37.0
WARNING: Skipping page https://storage.googleapis.com/jax-releases/jax_releases.html because the GET request got Content-Type: application/octet-stream.The only supported Content-Type is text/html
ERROR: Could not find a version that satisfies the requirement jaxlib==0.1.58 (from versions: 0.1.60, 0.1.61, 0.1.62, 0.1.63, 0.1.64, 0.1.65, 0.1.66, 0.1.67, 0.1.68, 0.1.69, 0.1.70, 0.1.71)
ERROR: No matching distribution found for jaxlib==0.1.58

What is stranger is that given the warning message, I am able to find that https://storage.googleapis.com/jax-releases/jax_releases.html has a content-type of html (as pip requires)

# matthewfeickert output
$ docker pull python:3.7
3.7: Pulling from library/python
Digest: sha256:6790dfb201d2fa06b0935ea0bfa1227d2240280af56d0fe08772b11eb31402b5
Status: Image is up to date for python:3.7
docker.io/library/python:3.7
$ docker run --rm -ti python:3.7 /bin/bash
root@f3a5030536d1:/# curl -is https://storage.googleapis.com/jax-releases/jax_releases.html | grep -i "^Content-Type:"
content-type: text/html
root@f3a5030536d1:/# 

but @kratsg running with the same commands in the same Docker container gets the erroneous content-type: application/octet-stream that the CI does

# kratsg output
$ docker pull python:3.7
3.7: Pulling from library/python
Digest: sha256:6790dfb201d2fa06b0935ea0bfa1227d2240280af56d0fe08772b11eb31402b5
Status: Image is up to date for python:3.7
docker.io/library/python:3.7
$ docker run --rm -ti python:3.7 /bin/bash
root@341e11194ff6:/# curl -is https://storage.googleapis.com/jax-releases/jax_releases.html | grep -i "^Content-Type:"
content-type: application/octet-stream

The only difference here is that we’re in geographically different locations — I’m in Illinois and @kratsg is in California — and so are hitting different servers.

Question: Could this mean that the content-type is not being set consistently across servers somehow?

cc @kratsg @lukasheinrich

[yashk2810]

https://cloud.google.com/storage/docs/xml-api/reference-headers#cachecontrol

Can you try this: curl -is https://storage.googleapis.com/jax-releases/jax_releases.html?ignoreCache=1 | grep -i "^Content-Type:"

[kratsg]

I guess in the future if we still have problems we can just use the ignoreCache version with --find-links

can use any query parameter e.g. ?foo=bar or ?random=12480123 since the caching is by URL.