extra-index-url doesn't work as suggested in "Using TestPyPI" guide

Environment

• pip version: 18.1
• Python version: 3.7.5
• OS: Linux, Ubuntu 19.10

Description I’m trying to test dependencies for my package in TestPyPI before uploading to “real” PyPI. The guide at: https://packaging.python.org/guides/using-testpypi/ says I can do this with: pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple your-package

So after uploading to TestPyPI with twine, I create a new virtualenv, activate it, pip install wheel, then: pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1 This fails with: Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/pycairo/

Which dependency it fails on varies; it’s not always pycairo, but the rest of the message is always the same. Note that it’s looking for dependencies in test.pypi.org, not pypi.org. If I just pip install pycairo (or whatever other dependency it fails on), that always works, so the packages are in pypi though not in test.pypi.

Expected behavior Based on the “Using TestPyPI” guide and the pip man page, I expected that this would look for pytopo version 1.6.1 in TestPyPI, find it, try to satisfy its dependencies, and when the dependencies weren’t found in TestPyPI it would look for them in PyPI, install them all then proceed to install pytopo from TestPyPI.

How to Reproduce

python3 -m venv /tmp/test3env
source /tmp/test3env/bin/activate
pip install wheel
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1

Output

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/pycairo/

[uranusjr]

One thing we probably can do is to raise/catch our own HTTP exceptions instead of requests.HTTPError if we can. This would mean to replace raise_for_exception() usages, and implement our own variant. It’s not particularly difficult, but I’m not sure whether it’d be worthwhile.

[uranusjr]

This looks very much like #6373. This HTTPError case should be caught if it originated from pip’s vendored requests copy (pip._vendor.requests), but your log shows pip is using a non-vendored copy instead (/tmp/test3env/share/python-wheels/requests-2.21.0-py2.py3-none-any.whl).

The problem here is Ubuntu messed up when de-bundling pip’s vendored dependencies. I don’t think there’s much pip can do at this time. Relevant Ubuntu bug report: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1833229