Build slows down from 7 minutes to 100+ minutes in contentful + Gatsby implementation

Description

We are experiencing extremely slow builds for Gatsby + Contentful. The issue arises sporadically when content is changed in contentful. (Using bitbucket pipelines). Builds usually take 5-8 minutes on pipelines, but for no apparent reason suddenly things start taking up 90+ minutes and it stays this way, until I revert all the changes to this point. Reverting makes the build fast again. Republishing the reverted changes, yields a quick build as well. So at the end there seems to be nothing particular causing the bug. The code between the two builds does not change and bitbucket pipelines run a clean install every time including all libraries etc. but suddenly it takes ages to compile.

I have no idea how to debug it.

We get warnings about slow queries sporadically too: warn Query takes too long: File path: /Users/artifex/Work/Projects/ln-website/src/templates/ContactPage.tsx URL path: /kontakt ....

Another weirdness is that a blank project (that I created for testing), that simply pulls in the data from the same contentful repository, but does not display any content and executes only one gql-query for one page, compiles fine and fairly quickly no matter what. So it seems to be related to the conjunction of code, data and build chain.

Some more info:

• We are using gatsby-plugin-graphql-codegen that generates a file that is about a 100mb in size and always gives warnings that it is deoptimized, due to big file size.
• The build seems to hang in the step “Building production JavaScript and CSS bundles”

This is the output for a build that runs as it should in about 7 minutes:

success createPages - 0.811s
success createPagesStatefully - 0.089s
success updating schema - 0.442s
success onPreExtractQueries - 0.002s
success extract queries from components - 2.127s
success write out redirect data - 0.002s
info [gatsby-plugin-graphql-codegen] definition for queries of schema default-gatsby-schema has been updated at graphql-types.ts
success Build manifest and related icons - 0.287s
success onPostBootstrap - 44.590s
info bootstrap finished - 89.415s
success run static queries - 1.381s - 11/11 7.96/s
success run page queries - 1.664s - 61/61 36.66/s
success write out requires - 0.005s
info Using stage environment
success created cachedVariables.json
success making environment variables available as globals
error [BABEL] Note: The code generator has deoptimised the styling of /opt/atlassian/pipelines/agent/build/graphql-types.ts as it exceeds the max of 500KB.
success Building production JavaScript and CSS bundles - 262.154s
success Rewriting compilation hashes - 0.003s
info Using stage environment
success created cachedVariables.json
success making environment variables available as globals
error [BABEL] Note: The code generator has deoptimised the styling of /opt/atlassian/pipelines/agent/build/graphql-types.ts as it exceeds the max of 500KB.
.
success Building static HTML for pages - 173.369s - 61/61 0.35/s
success Generating image thumbnails - 439.972s - 8/8 0.02/s
success onPostBuild - 0.002s
info Done building in 529.824 sec

This is the build that slows down to 90+ minutes

success createPages - 1.876s
success createPagesStatefully - 0.061s
success updating schema - 0.732s
success onPreExtractQueries - 0.001s
success extract queries from components - 3.152s
success write out redirect data - 0.001s
info [gatsby-plugin-graphql-codegen] definition for queries of schema default-gatsby-schema has been updated at graphql-types.ts
success Build manifest and related icons - 0.401s
success onPostBootstrap - 104.440s
info bootstrap finished - 198.792s
success run static queries - 2.828s - 11/11 3.89/s
success run page queries - 2.908s - 61/61 20.98/s
success write out requires - 0.009s
info Using stage environment
success created cachedVariables.json
success making environment variables available as globals
error [BABEL] Note: The code generator has deoptimised the styling of /opt/atlassian/pipelines/agent/build/graphql-types.ts as it exceeds the max of 500KB.
success Building production JavaScript and CSS bundles - 3334.661s
success Rewriting compilation hashes - 0.007s
info Using stage environment
success created cachedVariables.json
success making environment variables available as globals
error [BABEL] Note: The code generator has deoptimised the styling of /opt/atlassian/pipelines/agent/build/graphql-types.ts as it exceeds the max of 500KB.
.
success Building static HTML for pages - 2236.973s - 61/61 0.03/s
success Generating image thumbnails - 5580.829s - 8/8 0.00/s
success onPostBuild - 0.007s
info Done building in 5780.588 sec

Are there any ways to figure out what might be causing this issue and how to debug the process?

Steps to reproduce

This is incredibly hard to reproduce and I cannot give any access to our repositories, due to the commercial nature of the project. So hoping to get some help on debugging/figuring out what causes it.

Expected result

Build should run in a few minutes on pipelines.

Actual result

Build required 90+ minutes and uses insane amounts of memory.

Environment

Run gatsby info --clipboard in your project directory and paste the output here.

  System:
    OS: macOS 10.15.6
    CPU: (8) x64 Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 14.5.0 - /usr/local/bin/node
    npm: 6.14.5 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 85.0.4183.121
    Edge: 85.0.564.63
    Firefox: 73.0.1
    Safari: 13.1.2
  npmPackages:
    gatsby: ^2.24.2 => 2.24.2
    gatsby-background-image: ^1.1.1 => 1.1.1
    gatsby-env-variables: ^2.0.0 => 2.0.0
    gatsby-image: ^2.4.9 => 2.4.13
    gatsby-plugin-anchor-links: ^1.1.1 => 1.1.1
    gatsby-plugin-graphql-codegen: ^2.7.1 => 2.7.1
    gatsby-plugin-manifest: ^2.4.14 => 2.4.18
    gatsby-plugin-offline: ^3.2.18 => 3.2.18
    gatsby-plugin-page-transitions: ^1.0.8 => 1.0.8
    gatsby-plugin-react-helmet: ^3.3.6 => 3.3.10
    gatsby-plugin-sass: ^2.3.12 => 2.3.12
    gatsby-plugin-sharp: ^2.6.21 => 2.6.22
    gatsby-source-contentful: ^2.3.46 => 2.3.46
    gatsby-source-filesystem: ^2.3.14 => 2.3.19
    gatsby-transformer-remark: ^2.8.25 => 2.8.25
    gatsby-transformer-sharp: ^2.5.7 => 2.5.11
  npmGlobalPackages:
    gatsby-cli: 2.12.60

[lnilya]

TL;DR if anyone ever comes across it:

1. Remove links in your rich text fields
2. add _noBase64 for all your images.
3. if you are using “@contentful/gatsby-transformer-contentful-richtext”, don’t. just use “documentToReactComponents” instead and only pull the json from your rich text fields. => At least a 30% build performance increase for us.
4. uncomment “gatsby-plugin-graphql-codegen” if you are using it, and comment it back on, only on query changes. Also makes build about 30% faster.

If builds starts running again, add your images, and slowly start adding in the links. I am not a 100% sure, but the link issue might have been related to gatsby-plugin-graphql-codegen as well.

=> Make frequent backups of contentful via their cli:

//to pull out your data
contentful space export --space-id xxxxxxxxxx  

//to push it back in
contentful space import --space-id xxxxxxxxxx --environment-id xxx --content-file xxxxxx.json 

Thanks for all the support here!

[lnilya]

In search of a solution I disabled all base64 encoding by adding the _noBase64 to all image GatsbyContentfullFluid queries. It does not give the above mentioned error anymore, but the 2hr duration of the build still persists.