Adjust throttling for 429 response codes

HTTP 429 response code is returned when we reach a rate limit for an API at a given time. Usually, it is a matter of waiting a bit to start sending new requests. The “problem” is that, if concurrency settings are greater than the allowed number of requests from the API, we’ll always get 429s.

A solution would be to tune throttling so it delays requests based on 429s. It can be a extension/middleware, as AutoThrottle seems quite specific for throttle control over latency.

Also it could be worth considering that some APIs return the waiting time https://github.com/scrapy/scrapy/issues/3849

Here is a previous PR for this https://github.com/scrapy/scrapy/pull/3061

[caffeinatedMike]

@Gallaecio I think a per-domain basis would be more fitting for my use-case. However, with #5015 close to being approved (I’ve been eagerly following that discussion), I do believe that functionality will make it easier to account for status codes for throttling by subclassing the existing AutoThrottle middleware. Though, I feel this functionally would fall under a common dilemma, thus calling for an official middleware to address it.

[Gallaecio]

From the specification:

Note that this specification does not define how the origin server identifies the user, nor how it counts requests. For example, an origin server that is limiting request rates can do so based upon counts of requests on a per-resource basis, across the entire server, or even among a set of servers. Likewise, it might identify the user by its authentication credentials, or a stateful cookie.

So assuming that a 429 means “too many request to this domain”, it may mean “too many requests to this specific endpoint”, “too many requests with this cookie”, and so on. Maybe we could assume the domain scenario by default, since it is probably the most likely in web crawling, but ideally we should figure out a way to allow flexibility to deal with other scenarios effectively.