PyPI redirect loop for nonexistent (internal) package with hash

Bug description

Names were changed from the real name to “internal” due to my employer’s policy.

I have a requirements file like below:

(venv) $ cat requirements.txt 
internal==3.0.0 \
    --hash=sha256:abeb67f050c0f898333a1f6242c68ce4935863abab0f79edebb0e9611b01c8ff

Auditing this “internal” package on PyPI fails with an error:

(venv) $ pip-audit -s pypi -r requirements.txt  --index-url 'https://pypi.org/simple/'
ERROR:pip_audit._cli:PyPI is not redirecting properly
ERROR:pip_audit._cli:Tip: your network may be blocking this service. Try another service with `-s SERVICE`

I can successfully audit this non-public package using OSV:

(venv) $ pip-audit -s osv -r requirements.txt  --index-url 'https://pypi.org/simple/' --verbose
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Osv: 'osv'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple/', extra_index_urls=[], skip_editable=False)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.osv.dev:443
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2
No known vulnerabilities found

Version 2.1.1 of pip-audit gave a traceback:

(venv) $ pip-audit -s pypi -r requirements.txt  --index-url 'https://pypi.org/simple/' --verbose
Traceback (most recent call last):
  File "/redacted/venv/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/redacted/venv/lib/python3.9/site-packages/pip_audit/_cli.py", line 357, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/redacted/venv/lib/python3.9/site-packages/pip_audit/_audit.py", line 66, in audit
    for dep, vulns in self._service.query_all(specs):
  File "/redacted/venv/lib/python3.9/site-packages/pip_audit/_service/interface.py", line 143, in query_all
    yield self.query(spec)
  File "/redacted/venv/lib/python3.9/site-packages/pip_audit/_service/pypi.py", line 57, in query
    response: requests.Response = self.session.get(url=url, timeout=self.timeout)
  File "/redacted/venv/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/redacted/venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/redacted/venv/lib/python3.9/site-packages/requests/sessions.py", line 677, in send
    history = [resp for resp in gen]
  File "/redacted/venv/lib/python3.9/site-packages/requests/sessions.py", line 677, in <listcomp>
    history = [resp for resp in gen]
  File "/redacted/venv/lib/python3.9/site-packages/requests/sessions.py", line 166, in resolve_redirects
    raise TooManyRedirects('Exceeded {} redirects.'.format(self.max_redirects), response=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

Changing the name to “internaldne” instead of “internal” gives the expected result (a message stating the package cannot be found):

(venv) $ pip-audit -s pypi -r requirements.txt  --index-url 'https://pypi.org/simple/' --verbose
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple/', extra_index_urls=[], skip_editable=False)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/internaldne/3.0.0/json" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /pypi/internaldne/3.0.0/json HTTP/1.1" 404 24
DEBUG:cachecontrol.controller:Status code 404 not in (200, 203, 300, 301, 308)
DEBUG:pip_audit._service.pypi:Dependency not found on PyPI and could not be audited: internaldne (3.0.0)
No known vulnerabilities found
Name       Skip Reason
---------- -------------------------------------------------------------------------
internaldne Dependency not found on PyPI and could not be audited: internaldne (3.0.0)

Changing the hash to a random value does not change the result. Keeping the name as “internal”, but completely removing the hash in the requirements file, does make everything work:

(venv) $ pip-audit -s pypi -r requirements.txt  --index-url 'https://pypi.org/simple/' --verbose
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple/', extra_index_urls=[], skip_editable=False)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple//internal" in the cache                            
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple//internal HTTP/1.1" 404 13
DEBUG:cachecontrol.controller:Status code 404 not in (200, 203, 300, 301, 308)
DEBUG:pip_audit._dependency_source.resolvelib.resolvelib:Could not find project "internal" on any of the supplied index URLs: ['https://pypi.org/simple/']
No known vulnerabilities found
Name    Skip Reason
------- ------------------------------------------------------------------------------------------------
internal Could not find project "internal" on any of the supplied index URLs: ['https://pypi.org/simple/']

Auditing the “internal” package with a hash, but adding --verbose, shows pip-audit strips a trailing slash from the URL, and PyPI then redirects to the same URL with the trailing slash in a loop:

(venv) $ pip-audit -s pypi -r requirements.txt  --index-url 'https://pypi.org/simple/' --verbose
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple/', extra_index_urls=[], skip_editable=False)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/internal/3.0.0/json" in the cache
DEBUG:cachecontrol.controller:Returning cached permanent redirect response (ignoring date and etag information)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/internal/3.0.0/json/" in the cache
DEBUG:cachecontrol.controller:Returning cached permanent redirect response (ignoring date and etag information)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/internal/3.0.0/json" in the cache
DEBUG:cachecontrol.controller:Returning cached permanent redirect response (ignoring date and etag information)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/internal/3.0.0/json/" in the cache
DEBUG:cachecontrol.controller:Returning cached permanent redirect response (ignoring date and etag information)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/internal/3.0.0/json" in the cache
DEBUG:cachecontrol.controller:Returning cached permanent redirect response (ignoring date and etag information)
# And so on, until the TooManyRedirects exception is raised

Reproduction steps

Run pip-audit against the PyPI service. The requirements file should contain a package name and hash that is installed locally, but is not available on PyPI.

Expected behavior

Pip-audit reports that the package cannot be found, and continues auditing other packages.

Platform information

• OS name and version: Red Hat Enterprise Linux 8.5
• pip-audit version (pip-audit -V): 2.1.1 (gives traceback) and 2.3.3 (gives TooManyRedirects error)
• Python version (python -V or python3 -V): 3.9.7
• pip version (pip -V or pip3 -V): pip 21.3.1

Additional context

My employer has an internal PyPI mirror, but this package isn’t available there (only used by our team) and it gives the same behavior - TooManyRedirects.

[juspence]

Ah yes, forgot to add above - there’s no squatting as far as I can tell, running below all give the same result:

(venv) $ curl 'https://pypi.org/pypi/internal/3.0.0/json'
{"message": "Not Found"}
(venv) $ curl 'https://pypi.org/pypi/internaldne/3.0.0/json'
{"message": "Not Found"}
(venv) $ curl 'https://pypi.org/simple/internal/'
404 Not Found
(venv) $ curl 'https://pypi.org/simple/internaldne/'
404 Not Found

I’ll also be on PTO next week, so won’t be able to respond soon, but I will definitely take a look as soon as I’m back. Thanks for all your hard work on pip-audit - it’s a great tool and we’ve loved using it!

[woodruffw]

No problem! I’m glad it was a simple fix.

(Also, thank you for the kind words, and enjoy your PTO!)