Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

discord.py does not use Retry-After header for ratelimiting?

See original GitHub issue

Summary

Discord.py does not take into account Retry-After header when deciding to exhaust a per-endpoint bucket or sleep due to a rate limit.

Reproduction Steps

Become Cloudflare rate-limited on an endpoint (in my case, excessive calling to client.fetch_invite())
View the headers of the request, it includes a Retry-After with seconds until rate limiting will end

Example response returned (python logging.logger output at debug level) around when it occured:

2021-03-19 17:22:19,640 - discord.http - DEBUG - GET https://discord.com/api/v7/invites/XXXNDMI4hU1CqvFS with None has returned 404
2021-03-19 17:22:19,641 - DISCORD-CRAWLER __main__ - INFO - Code XXXNDMI4hU1CqvFS was not found
2021-03-19 17:22:19,641 - DISCORD-CRAWLER __main__ - INFO - 404 Not Found (error code: 10006): Unknown Invite
2021-03-19 17:22:19,645 - DISCORD-CRAWLER __main__ - INFO - Code XXXNDMI4hU1CqvFS previously queried at 1616188939
2021-03-19 17:22:19,645 - DISCORD-CRAWLER __main__ - INFO - Code XXXNDMI4hU1CqvFS previously queried at 1616188939
2021-03-19 17:22:19,646 - DISCORD-CRAWLER __main__ - INFO - Waiting 10 seconds before next query
2021-03-19 17:22:29,653 - DISCORD-CRAWLER __main__ - INFO - Fetching Discord invite for code XXXPIeMrgYLXtabq
2021-03-19 17:22:29,816 - discord.http - DEBUG - GET https://discord.com/api/v7/invites/XXXPIeMrgYLXtabq with None has returned 404
2021-03-19 17:22:29,816 - DISCORD-CRAWLER __main__ - INFO - Code XXXPIeMrgYLXtabq was not found
2021-03-19 17:22:29,817 - DISCORD-CRAWLER __main__ - INFO - 404 Not Found (error code: 10006): Unknown Invite
2021-03-19 17:22:29,820 - DISCORD-CRAWLER __main__ - INFO - Code XXXPIeMrgYLXtabq previously queried at 1616188949
2021-03-19 17:22:29,821 - DISCORD-CRAWLER __main__ - INFO - Waiting 10 seconds before next query
2021-03-19 17:22:39,825 - DISCORD-CRAWLER __main__ - INFO - Fetching Discord invite for code XXXpJtrjOinNVsrd
2021-03-19 17:22:39,843 - discord.http - DEBUG - GET https://discord.com/api/v7/invites/XXXpJtrjOinNVsrd with None has returned 429
2021-03-19 17:22:39,843 - DISCORD-CRAWLER __main__ - ERROR - Code XXXpJtrjOinNVsrd could not be retrieved!
2021-03-19 17:22:39,843 - DISCORD-CRAWLER __main__ - ERROR - 429 Too Many Requests (error code: 0): <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Access denied | discord.com used Cloudflare to restrict access</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" type="text/css" media="screen,projection" />


</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error hidden" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="p-0">
      <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased">
         <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight">
           <span data-translate="error">Error</span>
           <span>1015</span>
         </h1>
         <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">Ray ID: 6329c1e6ef25f21e &bull;</span>
         <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">2021-03-19 21:22:39 UTC</span>
        <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">You are being rate limited</h2>
      </header>

      <section class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
          <div id="what-happened-section" class="w-1/2 md:w-full">
            <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_happened">What happened?</h2>
            <p>The owner of this website (discord.com) has banned you temporarily from accessing this website.</p>
            
          </div>

          
      </section>

      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">6329c1e6ef25f21e</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: XX.XX.XX.50</span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>
    
  </p>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="text/javascript">
  window._cf_translation = {};
  
  
</script>

</body>
</html>

Expected Results

At the least, discord.py should wait until Retry-After header expires before querying again.

Ideally, it would apply the same logic to the per-endpoint ratelimitting bucket, as it does with the X-RateLimit-Remaining Header.

The ratelimitting I describe seems to only be on the one endpoint (invites)

Actual Results

discord.py will raise an HTTPException 429, but not mark the endpoint as exhausted, nor wait the designated amount of time.

Intents

None

System Information

Python v3.8.7-final
discord.py v1.6.0-final
aiohttp v3.7.4.post0
system info: Linux 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020

Checklist

I have searched the open issues for duplicates.
I have shown the entire traceback, if possible.
I have removed my token from display, if visible.

Prior-art:

This was kinda-sorta handled in here https://github.com/Rapptz/discord.py/issues/2035#issuecomment-478335139
Seems like there was something in here for it in like 2015, but I assume it was a much different time for Discord then… https://github.com/Rapptz/discord.py/blob/1e175f2ab3cb38c094b2eca8f9214d076b211c4b/discord/client.py#L773

Issue Analytics

State:
Created 3 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

Rapptzcommented, Mar 20, 2021

There is no millisecond precision with Retry-After. It’s rounded. So a ratelimit of 500ms is rounded up to 1 second.

I do not want to use Retry-After because it is imprecise. Rate limit headers return X-Ratelimit-Reset-After (which is what my snippet was showing). The only case where this isn’t returned is when there is a cloudflare ban. Cloudflare bans are not handled (and can’t be, because due to their nature they can last anywhere between 1 hour and multiple days). Waiting and sleeping for multiple hours is not desirable behaviour.

0reactions

Cobertoscommented, Mar 20, 2021

Would it be too kludge-y to divide by 1000 and add a padding of an extra 1000ms, just using the more imprecise time? I can make a PR if desired.

Otherwise, if that doesn’t work, this naive snippet would probably suffice for end-users:

#...
except discord.HTTPException as e:
  if e.code == 429 and 'Retry-After' in e.response.headers:
    secondsToSleep = int(e.response.headers['Retry-After'])
    await asyncio.sleep(secondsToSleep)