Credential prompt after --extra-index-url redirects not allowed by requests

Environment

• pip version: 19.0
• Python version: 3.6
• OS: Debian 8.11

Description

We have a private PyPI running, which has basic auth protection. The CI user has a ~/.config/pip/pip.conf file containing the --extra-index-url option with username & password in the URL for basic auth.

Since pip 19.0, there is a prompt for the username and password for this index URL, which wasn’t the case with 18.1. In a CI context where we cannot provide input, this is of course not convenient.

Expected behavior PIP/requests should use the credentials found in the URL in the pip.conf and not prompt for them.

How to Reproduce

1. Have a private PyPI
2. Configure this with basic auth username/password in the URL in pip.conf
3. pip install some-package - where some-package lives in the private PyPI

Output, from our CI

$ pip install -r requirements/jenkins.txt
User for <redacted>: Exception:
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 269, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 196, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 639, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 610, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 743, in _get_pages
    page = _get_html_page(location, session=self.session)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 229, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 177, in _get_html_response
    "Cache-Control": "max-age=0",
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/download.py", line 403, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 247, in resolve_redirects
    **adapter_kwargs
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 653, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/download.py", line 197, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)

[cjerdonek]

Great! 👍

I asked about redirects because pip’s vendored requests library was upgraded since the last release, and requests’s release notes mention a couple recent changes about how requests will now strip the authorization info in the presence of certain redirects like https -> http (which will show up as not supplying a username / pass – hence a prompt). And indeed, that’s what it looks like is happening from your logs.

[dstufft]

Unverified, but tagging as a 19.0 issue as this seems like a major regression.