Question: how to paginate with token? (or recursively call async function and pass through parameters)

AWS API calls typically respond with a pagination token. When present, another call to the same API should be made with that token, until a token is no longer supplied. This is typically done using recursion.

I would like to use async (or another library) to simplify this approach. I had imagined the following psuedo code:

  await paginate(async (lastToken, recurse, page) => {
    const { certificates, token } = await listCertificates({nextToken: lastToken})
    handleCertificates(certificates)
    token && recurse(token)
  })

This illustrates how something as complicated as token based pagination could be accomplished with a few lines of code. I’ve looked into async and saw some relevant functions (e.g. until), but have not found a way to recursively pass the output of one call to the next (i.e. pass in the token to the next call).

Is this possible with async? Or is there a library that’s more suitable for this?

[aearly]

doWhilst is good for this kind of thing.

let results = []
let nextToken
doWhilst(
  (next) => {
    getPage(url, nextToken, (err, body) => {
      if (err) return next(err);
      nextToken= body.nextToken
      results = results.concat(body.objects)
      next()
    })
  },
  () => nextToken,
  (err) => {
    // results now has all pages
  })

Although, with async/await, it’s easier to use a plain do/while:

let results = []
let nextToken
do {
  const body = await getPage(url, opts, nextToken)
  nextToken = body.nextToken
  results = results.concat(body.objects)
} while (nextToken)
// results now has everything

[tommedema]

@aearly I like how async/await makes something like do/while so simple to read again; thanks!