Always retry 429 Too Many Requests responses from index server

What’s the problem this feature will solve?

What I’m trying to do:

• As part of the bootstrapping process for nodes on our company’s compute cluster, I’m trying to use pip to install packages from our company’s private package index. Our private package index is hosted on AWS CodeArtifact.
• Our cluster can scale-out considerably and suddenly, creating a large spike in demand on our package index (in terms of number of requests) .
• As a result of the sudden spike in requests, our package index throttles the clients, by responding to some requests with 429 Too Many Requests (as is expected).

The problem I’m having:

• I would expect pip to retry these requests, since the response code is a common one to retry.
• Unfortunately, these requests are not retried, since our package index does not include the Retry-After header in the response, and pip doesn’t explicitly tell urllib3 to force-retry 429 responses, regardless of the absence of Retry-After.
• This means a single 429 response causes the entire pip install operation to abort.
• I fully acknowledge that, ideally, the server should include the Retry-After header in the response, and I’ve reached out to AWS to request this to be added. However, the header is not strictly required by RFC 6586.

Describe the solution you’d like

429 responses should be retried by pip, regardless of the absence of the Retry-After header. There should be no harm in doing so, as the status code indicates the client should simply wait a bit before retrying (it just doesn’t explicitly say how long to wait for).

This can be done by adding 429 as a status code in the status_forcelist of the pip network session’s urllib3.Retry config. This way, the --retries flag is also respected when it comes to the number of attempts on throttled requests.

Alternative Solutions

As mentioned above, I’ve reached out to AWS support to request that CodeArtifact includes the Retry-After header when throttling requests.

We will also be requesting the rate-limit buckets to be increased in capacity, so that the likelihood of being throttled is lower, however, 429 responses are an important part of HTTP communication, to force clients to spread the load on the server over time, so I don’t think this is a good long-term solution.

Additional context

All information in this ticket is regarding the latest stable release at the time of opening this ticket (22.0.4), and the request is agnostic to platforms or python versions.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

I seem to recall a similar discussion some time ago but can’t find the issue right now. I think pip should not retry 429 Too Many Requests. A server responds this because the client has sent too many requests, so pip should “resolve” it by stop sending any more, not to retry and sending even more requests.

[uranusjr]

Pip already retries 500, 503, 520, 527 errors without issuing a warning, is there any particular reason why 429 warrants a warning more than these other status codes?

I don’t know the exact answer (the choice was made long ago and those maintainers are not around), but this choice makes sense to me. A 4xx status indicates user error, and a warning conveys a potential action item; a 5xx indicates a server error the user generally can’t do anything about, and a warning in this case adds more frustration than purpose.