Rate limiting

I have a Node.js app which tries to upload more then 30 records a second. Airtable API doesn’t allow us to do that. But this way the retry logic dies after ~3 mins for me.

I’ve solved it for myself with RateLimiter

Currently it looks something like this:

const RateLimiter = require('limiter').RateLimiter;
const limiter = new RateLimiter(5, 'second');

// ...

limiter.removeTokens(1, () => base('TAB').create({ ... }, err => err ? console.log(err) : void 0));

What if we implemented this in airtable.js?

[coreyward]

Yeah I’d love to use Airtable as a source with Gatsby, but the rate limiting is rather restrictive and limiting. The way Gatsby works is to fetch an entire dataset, build a GraphQL schema using inferred types from that data, and then all queries (during development or build) run against the local GraphQL instance. So a flurry of requests, then nothing until another build. As it stands, requesting a handful of records can take several minutes due to rate-limit penalties.

While improving the rate limiting logic in this library would be nice, it’s really just treating a symptom. The real issue is the petty 30-second penalty for exceeding 5rps (which a web browser does every single time it loads a website: there are typically 6 runner threads running). At the very least Airtable could adjust the rate limiting to operate over a larger interval (e.g. 50 requests over a rolling 10 second window). This would better accommodate short bursts of activity, and would probably force the whoever is tasked with the the work to reimplement this using a window so that a timeout penalty isn’t even necessary.

[Solinx]

I’d actually love to see AirTable implement a more robust rate limiting that allows for short bursts.

We’re going to pull a data from tables to generate a few static JSON files that are going to be used by our application as data source. For a few seconds we’re make a load of calls, and then for hours or even days it’s going to be 0.