proposal: replace keyring dependency with generic pip-defined credential helper API

Currently pip directly imports keyring in order to use it for authentication. Furthermore, to avoid a performance hit in the majority case, it will only consult keyring if the original request fails with 401. This has caused a few issues:

1. Some users do not want pip to use keyring, even if it is present on the system. #8719
2. When interacting with a private PyPI repo, the initial unauthenticated request is undesirable. #10269
3. Due to a missing keyring feature (at least on macos), the username still has to be specified in the index url. This in turn causes pip’s credential cache to not function properly. #10288
4. Relying on keyring leads to a chicken-or-the-egg bootstrapping issue. You need pip in order to install keyring, but if you need to auth with a private PyPI repo to do so, now what?
5. In general, having pip directly rely on a specific external module seems undesirable. What happens if keyring falls out of vogue, or is abandoned?

I would like to propose that the direct dependency on keyring be removed, and replaced with a more generic concept, akin to docker’s credential helper concept. https://docs.docker.com/engine/reference/commandline/login/#credentials-store

The basic idea is that the pip configuration be able to specify a credential helper library for a given PyPI hostname. If there is no entry for a hostname, then requests are sent without any authentication. This allows the majority case of talking to public PyPI to continue to work as it does today with no performance hit. If a hostname does have an entry, then it is expected to map to a library that exposes a get_pip_credentials(hostname) function, which takes the download hostname and returns the username and password to use. The user just needs to arrange for this library to be importable, so for example they could write a custom library and add it to their PYTHONPATH.

For keyring (excluding macos), a very simple adapter around the keyring API could be created, and ideally could even be part of keyring itself. For macos, I can make my own shim to pull the keyring username from some environment variable of my choosing. For CI/CD, a simple credential helper could be made to pull the username and password from pre-canned environment variables. (Since this logic lives entirely outside pip, the issue of multiple index urls mentioned here is not relevant.)

If a user doesn’t want to use keyring, they simply would not specify it in their pip configuration. That means that by default, keyring would never be used at all. But it also means that users that do want to use it are not required to pass some flag to pip all the time.

If the credential helper API is standardized, then it can be shared with other dependency management tools such as pipenv, instead of the situation today where the two have completely different approaches. (For reference, most docker-like tools, such as skopeo and buildkit, will abide by your credential helpers, thus providing a consistent experience.)

[pfmoore]

… on the other hand, if we use an entry point, someone could write a helper (in Python) that did all of that config management, delegation to arbitrary executables, dispatching based on domain, etc.

That saves us having to get into the business of designing credential management schemes, and lets interested parties come up with their own mechanisms. I’d expect “standard” helpers to get developed and published, so this wouldn’t require everyone to write their own.

(Basically, I’m arguing that we have the bare minimum in pip, and let others do the work 😉)

[sbidoul]

allowing non-Python based implementations

a small python wrapper could call non-python implementations ?

doesn’t require them to manage duplicated configuration stuff

fair

dispatch to different credential managers based on the domain

credential managers could accept the domain as argument, and pip could try them all until one knows about the domain ?

OTOH returning a user and password is only one possible authentication mechanism (people have been asking for ways to inject http headers etc). So the authentication and possibly in the future the http session mechanisms are bound to grow in complexity, which may or may not make them difficult to handle via pip’s configuration.