Add support for auto pagination when fetching results

Hi, I’m currently working on a project where this would be useful. For each method that fetches paginated results, it’d be nice if there existed a convenience method that fetched all the pages in parallel.

Take getMySavedTracks for example, which has a maxinum number of 50 results per page. We’d have a method called:

getAllMySavedTracks(maxPages, optionsEach, callbackEach, callback)

Which fetches the first page, checks the total items, and creates requests if needed for each page using a limit and offset. The parameters could work as follows:

• maxPages: The maximum number of pages to get. By default it would request all of them (unless there’s a rate limit for pages Spotify imposes that I don’t know of).
• optionsEach: The object for options to send in each call to getMySavedTracks. Passing limit and offset would have no effect since that’s automatically determined per each page request in this method.
• callbackEach: A callback function for each page request i.e. each call to getMySavedTracks. The callback function receives the error and value parameters of course.
• callback: A callback function that fires after completion of all requests. Just going off of how Promise.all() works, you could pass it the array of values from all the resolved getMySavedTracks promises in the second parameter, and If any of the page requests fail, you only get the value of the promise that was rejected in the first parameter.

Currently, I do this on my own and it gets messy fast. Putting it in the API wrapper could save developers some headaches. If the paging could somehow be generalized, you’d end up with several clean convenience methods. The main usecase for this is for when dealing with a user who has a large music collection (personally I hoard and create too many playlists).

Let me know what you think. I’m totally willing to implement this, as I already have existing code I could refine to work within the wrapper.

[omarryhan]

I made this Typescript auto pagination helper. Unfortunately it doesn’t run in parallel, but I hope some would find it useful:

interface Pagination {
  next?: string;
  items: object[];
}

export const getAllPages = async <Response extends Pagination>(
  request: Promise<Response>,
): Promise<Response> => {
  const paginatedResponse = await request;

  let currentResponse = paginatedResponse;

  while (currentResponse.next) {
    currentResponse = await spotifyApi.getGeneric(
      currentResponse.next,
    ) as Response;
    paginatedResponse.items = paginatedResponse.items.concat(currentResponse.items);
  }

  return paginatedResponse;
};

To call it:

const paginatedResponse = await getAllPages<SpotifyApi.PlaylistTrackResponse>(
  spotifyApi.getPlaylistTracks(playlistId, { limit: 50 }),
);

[JMPerez]

Hi @mchateloin,

This sounds really interesting. I have thought of that in the past and it all depends on how smart you want the wrapper to be. At the moment it is just a mapping 1:1 with the Web API, which works well for most cases except for the ones you mentioned.

I already had a similar situation when coding the Spotify Deduplicator. In the end I would fetch the first page of items to get the length of the collection, and then would generate promises for the rest of the pages.

The main issue with having methods returning so many items is that they hide the undergoing requests. Due to the rate limiting of the Web API, one can easily exceed it when fetching all user’s saved tracks or all user’s playlists. Actually, that is what happened with certain users with a very large collection of music. So I decided to throttle the promises, delaying them a little bit to make sure I wasn’t making too many requests, following an approach which I moved later to its own package.

In the end I thought that the cases in which one would have to traverse many pages in a collection would be few, and I wanted for the developer to be able to set a certain rate limit, so the wrapper wouldn’t blindly make a ton of requests.

I think a good start is that you create something on top of the wrapper that can generically manage pagination and get X results or all results. And internally it would handle queueing the requests and managing rate limiting.