Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

I have a Node.js app which tries to upload more then 30 records a second. Airtable API doesn’t allow us to do that. But this way the retry logic dies after ~3 mins for me.

I’ve solved it for myself with RateLimiter

Currently it looks something like this:

const RateLimiter = require('limiter').RateLimiter;
const limiter = new RateLimiter(5, 'second');

// ...

limiter.removeTokens(1, () => base('TAB').create({ ... }, err => err ? console.log(err) : void 0));

What if we implemented this in airtable.js?

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:12
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

coreywardcommented, Feb 24, 2019

Yeah I’d love to use Airtable as a source with Gatsby, but the rate limiting is rather restrictive and limiting. The way Gatsby works is to fetch an entire dataset, build a GraphQL schema using inferred types from that data, and then all queries (during development or build) run against the local GraphQL instance. So a flurry of requests, then nothing until another build. As it stands, requesting a handful of records can take several minutes due to rate-limit penalties.

While improving the rate limiting logic in this library would be nice, it’s really just treating a symptom. The real issue is the petty 30-second penalty for exceeding 5rps (which a web browser does every single time it loads a website: there are typically 6 runner threads running). At the very least Airtable could adjust the rate limiting to operate over a larger interval (e.g. 50 requests over a rolling 10 second window). This would better accommodate short bursts of activity, and would probably force the whoever is tasked with the the work to reimplement this using a window so that a timeout penalty isn’t even necessary.

Solinxcommented, Jan 15, 2019

I’d actually love to see AirTable implement a more robust rate limiting that allows for short bursts.

We’re going to pull a data from tables to generate a few static JSON files that are going to be used by our application as data source. For a few seconds we’re make a load of calls, and then for hours or even days it’s going to be 0.

Read more comments on GitHub >

github_iconTop Results From Across the Web

What is rate limiting? | Rate limiting and bots - Cloudflare
Rate limiting is a strategy for limiting network traffic. It puts a cap on how often someone can repeat an action within a...
Read more >
Rate limiting - Wikipedia
In computer networks, rate limiting is used to control the rate of requests sent or received by a network interface controller.
Read more >
Rate-limiting strategies and techniques - Google Cloud
Rate limiting refers to preventing the frequency of an operation from exceeding some constraint. In large-scale systems, rate limiting is ...
Read more >
What is Rate Limiting | Types & Algorithms - Imperva
Rate limiting is a technique to limit network traffic to prevent users from exhausting system resources. Rate limiting makes it harder for malicious...
Read more >
Rate Limiter: Explained | Built In
Rate limiting refers to preventing the frequency of an operation from exceeding a defined limit. In large-scale systems, rate limiting is ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found