gatsby-source-contentful Response Size Too Big

Description

We’re running into an issue similar to #20081 with gatsby-source-contentful where once we have too many entries, we’re getting an error “400 Response size too big. Maximum allowed response size: 7340032b.” I’ve run into the same error with contentful CLI which can be mitigated with --max-allowed-limit, but there doesn’t seem to be any pagination limit options in gatsby-source-contentful.

Steps to reproduce

Run gatsby develop or gatsby build

Expected result

The site should build

Actual result

Error: Request failed with status code 400

Environment

Run gatsby info --clipboard in your project directory and paste the output here.

System: OS: macOS 10.14.6 CPU: (8) x64 Intel® Core™ i7-4870HQ CPU @ 2.50GHz Shell: 3.0.2 - /usr/local/bin/fish Binaries: Node: 13.1.0 - /usr/local/bin/node Yarn: 1.19.1 - /usr/local/bin/yarn npm: 6.12.1 - /usr/local/bin/npm Languages: Python: 2.7.16 - /usr/local/bin/python Browsers: Chrome: 79.0.3945.130 Firefox: 70.0.1 Safari: 13.0.5 npmPackages: gatsby: ^2.19.1 => 2.19.1 gatsby-cli: ^2.8.2 => 2.8.19 gatsby-image: ^2.1.4 => 2.2.36 gatsby-plugin-canonical-urls: ^2.0.12 => 2.1.18 gatsby-plugin-catch-links: ^2.0.13 => 2.1.21 gatsby-plugin-feed: ^2.2.3 => 2.3.25 gatsby-plugin-google-tagmanager: ^2.1.24 => 2.1.24 gatsby-plugin-i18n: ^1.0.1 => 1.0.1 gatsby-plugin-manifest: ^2.2.33 => 2.2.33 gatsby-plugin-netlify: ^2.0.15 => 2.1.30 gatsby-plugin-offline: ^2.1.3 => 2.2.10 gatsby-plugin-react-helmet: ^3.0.12 => 3.1.18 gatsby-plugin-s3: ^0.3.2 => 0.3.2 gatsby-plugin-schema-snapshot: ^1.0.0 => 1.0.0 gatsby-plugin-sharp: ^2.1.5 => 2.3.8 gatsby-plugin-sitemap: ^2.0.12 => 2.2.24 gatsby-plugin-styled-components: ^3.0.7 => 3.1.16 gatsby-remark-autolink-headers: ^2.0.16 => 2.1.21 gatsby-remark-images-contentful: ^2.0.10 => 2.1.27 gatsby-remark-responsive-iframe: ^2.2.10 => 2.2.30 gatsby-source-contentful: ^2.1.55 => 2.1.70 gatsby-source-filesystem: ^2.0.39 => 2.1.42 gatsby-transformer-remark: ^2.3.10 => 2.6.42 gatsby-transformer-sharp: ^2.1.21 => 2.3.9 npmGlobalPackages: gatsby-cli: 2.7.30

[deriegle]

I tried my hand at putting in a fix for this. You can see that here. I can see this being useful for some of the projects I’m working on with Gatsby <> Contentful integrations.

[alana314]

@yomete contentful has added a “limit” parameter to their sync API! Here’s what I did to get it working: In gatsby-source-contentful/src/fetch.js line 78, added limit: 10: let query = syncToken ? { nextSyncToken: syncToken } : { initial: true, limit: 10 } (later I would make this a plugin option, and include tests) In contentful/lib/paged-sync.js line 139, added delete query.limit, since it doesn’t accept a limit after the initial query:

if (query.sync_token) {
    delete query.limit
    delete query.initial
    delete query.type
    delete query.content_type
  }

I added a pull request for contentful:
https://github.com/contentful/contentful.js/pull/374