Help with Authentication when proxying through CDN

Overview

I recently wrote an article on using Airtable (Airtable.js + lambda) as a content backend. https://medium.com/@gtalarico/using-airtable-as-a-content-backend-e373cd0d9974

A few people mentioned the 5 req/sec rate limit could be an issue, so I am trying to add additional details about how one could add a CDN (eg. Cloudfront) to cache the requests and avoid the rate limit.

I created airtable as the origin on cloudfront, and set it up to pass through Authorization headers. I am able to successful use the CDN to proxy/cache the calls

Request works as expected through curl:

$ curl -L "https://d2j5hchu5g3uxq.cloudfront.net/v0/appNtnZ99fkL1cByn/entries?maxRecords=100&view=all" \
    -H "authorization: Bearer keyOmitted"

STATUS: 200

{"records":[{"id":"recW0FpRZMaQv5Ble","fields":{"title":"3D 
[... truncated ]

Requests fail when using airtable.js

I then make the exact same request When I make the same call through airtable.js, but I get AUTHENTICATION_REQUIRED.

I logged the options variable right before airtable.js runs runAction and the payload looks good and matches my curl, so I really can’t understand why it would fail:

// run_action.js
# line 52
console.log(JSON.stringify(options))
request(options, function(error, resp, body) {
[ ... ]

Outputs:

// `options`
{
   "method":"GET",
   "url":"https://d2j5hchu5g3uxq.cloudfront.net/v0/appNtnZ99fkL1cByn/entries?maxRecords=100&view=all",
   "json":true,
   "timeout":300000,
   "headers":{
      "authorization":"Bearer keyOMITTED",
      "x-api-version":"0.1.0",
      "x-airtable-application-id":"appNtnZ99fkL1cByn",
      "User-Agent":"Airtable.js/0.1.0"
   },
   "agentOptions":{
      "rejectUnauthorized":false
   }
}

Response with status 500 in 686 ms.
Error during invocation:  s {
  error: 'AUTHENTICATION_REQUIRED',
  message: 'You should provide valid api key to perform this operation',
  statusCode: 401
}

I apologize as this may not be an issue with the lib but I am out of ideas here. Let me know if you have any insights, otherwise feel free to close it.

Here is section I am adding to the article - I would be happy to add a wiki page or doc so other users can use the same approach to cache the rate limit:

Article Update:

[gtalarico]

After some more tests and tweaking CDN settings I was able to get it to work. I am not 100% what the change was that fixed it, but I will post the setting here as a reference.

Thank you @EvanHahn and @syrnick for you help troubleshooting this.

Distribution Config

Behaviour Config

Origin Config

Netlify Lambda Code

const Airtable = require('airtable')

Airtable.configure({
  // AIRTABLE_API_URL="https://d2j5hchu5g3uxq.cloudfront.net"
  endpointUrl: process.env.AIRTABLE_API_URL,
  apiKey: process.env.AIRTABLE_API_KEY
})
const base = Airtable.base('appNtnZ99fkL1cByn')

exports.handler = function(event, context, callback) {
  const allRecords = []
  base('entries')
    .select({
      maxRecords: 100,
      view: 'all'
    })
    .eachPage(
      function page(records, fetchNextPage) {
        records.forEach(function(record) {
          allRecords.push(record)
        })
        fetchNextPage()
      },
      function done(err) {
        if (err) {
          callback(err)
        } else {
          const body = JSON.stringify({ records: allRecords })
          const response = {
            statusCode: 200,
            body: body,
            headers: {
              'content-type': 'application/json',
              'cache-control': 'Cache-Control: max-age=300, public'
            }
          }
          callback(null, response)
        }
      }
    )
}

[ricricucit]

ah! magic for me, as well. It started working after some time, and after invalidating all objects.

Not sure, but I think it was only cloudfront needing some time to go up, properly. 🤔