22.1: netrc file is only parsed when challenged with HTTP/401

Description

The change in NETRC handling per https://github.com/pypa/pip/issues/10979 may have introduced an unexpected side-effect.

https://github.com/pypa/pip/compare/22.0.4...22.1b1?diff=split

In the current release, netrc is only parsed when the PyPI repo returns an HTTP/401 error due to a change in the method signature’s default value from True to False:

    def _get_new_credentials(
        self,
        original_url: str,
        *,
        allow_netrc: bool = False,
        allow_keyring: bool = False,
    ) -> AuthInfo:

And further:

    def handle_401(self, resp: Response, **kwargs: Any) -> Response:

        # Query the keyring for credentials:
        username, password = self._get_new_credentials(
            resp.url,
            allow_netrc=False,
            allow_keyring=True,
        )

However, some PyPI repositories (such as those by JFrog Artifactory) can return HTTP/404 when auth isn’t presented, and HTTP/200 otherwise, like so:

GET /api/pypi/pypi-private/simple/my-awesome-lib/ HTTP/1.1
Host: artifacts.example.com
User-Agent: pip/22.1 {"ci":null,"cpu":"x86_64","distro":{"name":"macOS","version":"12.4"},"implementation":{"name":"CPython","version":"3.10.3"},"installer":{"name":"pip","version":"22.1"},"openssl_version":"OpenSSL 1.1.1n  15 Mar 2022","python":"3.10.3","setuptools_version":"58.1.0","system":{"name":"Darwin","release":"21.5.0"}}
Accept-Encoding: gzip, deflate
Accept: text/html
Connection: keep-alive
Cache-Control: max-age=0

HTTP/1.1 404 
Server: ArtifactoryHttpServer/47a3fd1d6c202c53658865bf00439b03f16f48e4
Date: Thu, 12 May 2022 04:56:57 GMT
Content-Type: application/json
Transfer-Encoding: chunked
X-JFrog-Version: Artifactory/7.37.14 73714900
X-Artifactory-Id: 090ed27bcfd2435b77f88876e97cddc25c8604f6
Content-Encoding: gzip
X-Proxy-Cache-Status: HIT
X-Proxy-Cache-Status: EXPIRED
X-B3-TraceId: 6e3a84b1422a9e79
Strict-Transport-Security: max-age=31536000
Connection: keep-alive

Whereas in 22.0.4:

GET /api/pypi/pypi-private/simple/my-awesome-lib/ HTTP/1.1
Host: artifacts.example.com
User-Agent: pip/22.0.4 {"ci":null,"cpu":"x86_64","distro":{"name":"macOS","version":"12.4"},"implementation":{"name":"CPython","version":"3.10.3"},"installer":{"name":"pip","version":"22.0.4"},"openssl_version":"OpenSSL 1.1.1n  15 Mar 2022","python":"3.10.3","setuptools_version":"58.1.0","system":{"name":"Darwin","release":"21.5.0"}}
Accept-Encoding: gzip, deflate
Accept: text/html
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Basic ****REDACTED****

HTTP/1.1 200 
Server: ArtifactoryHttpServer/47a3fd1d6c202c53658865bf00439b03f16f48e4
Date: Thu, 12 May 2022 05:13:08 GMT
Content-Type: text/html
Transfer-Encoding: chunked
X-JFrog-Version: Artifactory/7.37.14 73714900
X-Artifactory-Id: 090ed27bcfd2435b77f88876e97cddc25c8604f6
Cache-Control: max-age=300
Content-Encoding: gzip
X-Proxy-Cache-Status: EXPIRED
X-Proxy-Cache-Status: MISS
X-B3-TraceId: 2dbc36045a554171
Strict-Transport-Security: max-age=31536000
Connection: keep-alive

Expected behavior

Credentials from netrc are used at the first request

pip version

22.1

Python version

3.10.3

OS

macOS 12.4

How to Reproduce

1. pip install --upgrade pip==22.1
2. Create a ~/.netrc file
3. pip install some-package --index-url=https://artifacts.example.com/simple

Assumptions:

• PyPI server returns different results depending on when auth is first presented or not

Output

~/ $ pip3 install --debug --verbose my-awesome-lib --index-url=https://artifacts.example.com/api/pypi/pypi-private/simple --no-cache-dir
Using pip 22.1 from /Users/gledesma/.pyenv/versions/3.10.3/lib/python3.10/site-packages/pip (python 3.10)
Looking in indexes: https://artifacts.example.com/api/pypi/pypi-private/simple
ERROR: Could not find a version that satisfies the requirement my-awesome-lib (from versions: none)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Let’s revert #10998 then – we can figure out the proper fix for it in a follow up. 😃

[pradyunsg]

Can someone file a PR for this?