gatsby-source-contentful rate limit preview endpoint

Description

When running gatsby build pointed to contentful preview, we get many (1000+) API errors about rate limits.

Steps to reproduce

We have a large website. When we run gatsby build we see thousands of these lines:

[GraphQL error]: Message: An unexpected error happened on our side., Location: [object Object], Path: preview
[GraphQL error]: Message: An unexpected error happened on our side., Location: [object Object], Path: preview
[GraphQL error]: Message: An unexpected error happened on our side., Location: [object Object], Path: preview
[GraphQL error]: Message: An unexpected error happened on our side., Location: [object Object], Path: preview
[GraphQL error]: Message: An unexpected error happened on our side., Location: [object Object], Path: preview
[GraphQL error]: Message: You have exceeded the rate limit of the Organization this Space belongs to by making too many API requests within a short timespan. Please wait a moment before trying the request again., Location: undefined, Path: undefined
[Network error]: ServerError: Response not successful: Received status code 429
[GraphQL error]: Message: You have exceeded the rate limit of the Organization this Space belongs to by making too many API requests within a short timespan. Please wait a moment before trying the request again., Location: undefined, Path: undefined
[Network error]: ServerError: Response not successful: Received status code 429
[GraphQL error]: Message: You have exceeded the rate limit of the Organization this Space belongs to by making too many API requests within a short timespan. Please wait a moment before trying the request again., Location: undefined, Path: undefined
[Network error]: ServerError: Response not successful: Received status code 429

Occasionally, when we run builds pointed at either cdn.contentful.com or preview.contentful.com, we also get this error:

success createSchemaCustomization - 0.057s
error [object Object] Cannot read property 'content-length' of undefined
  TypeError: Cannot read property 'content-length' of undefined
  
  - fetch.js:31 responseLogger
    [gatsby-project]/[gatsby-source-contentful]/fetch.js:31:46
  
  - index.es-modules.js:217 
    [gatsby-project]/[contentful]/[contentful-sdk-core]/dist/index.es-modules.js:217:1
  
  - runMicrotasks
  
  - task_queues.js:97 processTicksAndRejections
    internal/process/task_queues.js:97:5
  
  - fetch.js:119 contentfulFetch
    [gatsby-project]/[gatsby-source-contentful]/fetch.js:119:23
  
  - gatsby-node.js:200 Object.exports.sourceNodes
    [gatsby-project]/[gatsby-source-contentful]/gatsby-node.js:200:9
  
  - api-runner-node.js:460 runAPI
    [gatsby-project]/[gatsby]/src/utils/api-runner-node.js:460:16
  
not finished source and transform nodes - 8.287s
not finished Contentful: Fetch data ([project id]) - 8.216s
error Command failed with exit code 1.

We believe this is related to hitting the organizational API limit via preview above.

Expected result

Gatsby should build

Actual result

Gatsby fails building. Sometimes preview succeeds after 1000s of the 429 errors and sometimes it fails. If we see the content-length error from fetch.js in gatsby-source-contentful it always fails.

Environment

System: OS: macOS 11.1 CPU: (12) x64 Intel® Core™ i7-9750H CPU @ 2.60GHz Shell: 3.1.2 - /usr/local/bin/fish Binaries: Node: 13.12.0 - /usr/local/bin/node Yarn: 1.22.10 - /usr/local/bin/yarn npm: 6.14.4 - /usr/local/bin/npm Languages: Python: 2.7.16 - /usr/bin/python Browsers: Chrome: 87.0.4280.88 Firefox: 84.0.1 Safari: 14.0.2 npmPackages: gatsby-core-utils: ^1.7.0-next.0 => 1.7.0-next.0 gatsby-plugin-utils: ^0.6.0-next.0 => 0.6.0-next.0 gatsby-source-filesystem: ^2.8.0-next.0 => 2.8.0-next.0 npmGlobalPackages: gatsby-cli: 2.7.30

[petergray-maker]

We are working on resolving these issues. Stay tuned 😃

Thanks, I got the same issue with 429 errors from contentful, would be great to handle these

[axe312ger]

We are working on resolving these issues. Stay tuned 😃