Really slow build times when adding locales in Contentful

Description

We have a Gatsby + Contentful project with 7 locales. The build process for this project is slowing to a crawl as we add content in the CMS. I created a local plugin that printed the “number of nodes by type” generated during Gatsby build. I was able to determine each added locale acts as a multiplier for the amount of nodes generated in Gatsby. So, for example, if there are 1000 assets in Contentful you get 7000 Gatsby nodes (for 7 locales) despite several assets not being available in all locales.

Steps to reproduce

There isn’t a clear process to reproduce this without having access to a Contentful space that can have multiple locales. If you do, then the following gatsby-node.js hook can help debug the steep increase in nodes as you enable more locales:

module.exports = {
  onPostBootstrap: args => {
    const nodes = args.getNodes();

    const counts = nodes.reduce((acc, node) => {
      const type = node.internal.type;
      const count = (acc[type] || 0) + 1;
      return { ...acc, [type]: count };
    }, {});

    console.table({ ...counts, Total: nodes.length });
  }
}

Expected result

I’m not sure if there is a broad stroke approach to avoid duplicating unused nodes for a given locale across all Contentful content types but at least for the case of assets it is entirely possible (and I guess, valid) to avoid generating ContentfulAsset nodes where an asset is unavailable in a given locale e.g. checking the file field is not null for a ContentfulAsset node.

Actual result

With 6 locales:

  success source and transform nodes - 41.782s
+ success building schema - 280.739s
  success createPages - 11.840s
  success createPagesStatefully - 0.081s
  success onPreExtractQueries - 0.003s
  success update schema - 0.292s
  success extract queries from components - 2.789s
  success write out requires - 0.134s
  success write out redirect data - 0.003s
  success Build manifest and related icons - 0.087s
  success onPostBootstrap - 3.946s
+ info bootstrap finished - 363.509 s

+ Total nodes: 55669

Breakdown of node counts by type: with-6-locales.txt

With 7 locales:

  success source and transform nodes - 50.208s
- success building schema - 632.450s
  success createPages - 20.045s
  success createPagesStatefully - 0.093s
  success onPreExtractQueries - 0.005s
  success update schema - 0.368s
  success extract queries from components - 3.301s
  success write out requires - 0.191s
  success write out redirect data - 0.003s
  success Build manifest and related icons - 0.077s
  success onPostBootstrap - 5.728s
- info bootstrap finished - 735.300 s

- Total nodes: 64671

Breakdown of node counts by type: with-7-locales.txt

Environment

Contentful stats:

• 3019 entries
• 1769 assets
• 41 content types
• 6 locales (we want to add a 7th)

Gatsby environment:

  System:
    OS: macOS Mojave 10.14.6
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Shell: 5.7.1 - /usr/local/bin/zsh
  Binaries:
    Node: 12.13.0 - /var/folders/11/zcvk9j4j7rd_dj0t5hcv9wc80000gp/T/yarn--1576691369326-0.6597870373283115/node
    Yarn: 1.21.1 - /var/folders/11/zcvk9j4j7rd_dj0t5hcv9wc80000gp/T/yarn--1576691369326-0.6597870373283115/yarn
    npm: 6.13.2 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 79.0.3945.88
    Firefox: 71.0
    Safari: 13.0.4
  npmPackages:
    gatsby: 2.18.13 => 2.18.13 
    gatsby-cli: ^2.8.19 => 2.8.19 
    gatsby-image: ^2.2.37 => 2.2.37 
    gatsby-plugin-catch-links: ^2.1.21 => 2.1.21 
    gatsby-plugin-feed: ^2.3.25 => 2.3.25 
    gatsby-plugin-manifest: ^2.2.33 => 2.2.33 
    gatsby-plugin-react-helmet: ^3.1.18 => 3.1.18 
    gatsby-plugin-sharp: ^2.3.9 => 2.3.9 
    gatsby-plugin-sitemap: ^2.2.24 => 2.2.24 
    gatsby-plugin-svgr: ^2.0.2 => 2.0.2 
    gatsby-plugin-typescript: ^2.1.23 => 2.1.23 
    gatsby-plugin-webpack-bundle-analyzer: ^1.0.5 => 1.0.5 
    gatsby-remark-prismjs: ^3.3.27 => 3.3.27 
    gatsby-source-contentful: ^2.1.71 => 2.1.71 
    gatsby-source-filesystem: ^2.1.42 => 2.1.42 
    gatsby-transformer-remark: ^2.6.43 => 2.6.43 
    gatsby-transformer-sharp: ^2.3.9 => 2.3.9 
  npmGlobalPackages:
    gatsby-cli: 2.7.15

[disintegrator]

@pvdz thanks for the feedback 😃 Just a couple of comments:

If I’m reading it right, that’s difficult. One node isn’t another and without an explicit schema we don’t know whether one node is the same as others without going through the whole thing explicitly. And that’s super expensive, unfortunately.

What I was suggesting is that gatsby-source-contentful introduce an explicit type ContentfulRichText @dontInfer type using createSchemaCustomization and assign that to all rich text fields it encounters in the content. My guess is that schema inference code would no longer need to walk down those fields in contentful nodes and output such a large amount of redundant types. Alternatively, the source plugin should use the same type for all rich text nodes it encounters. This corresponds to the following change in normalize.js:

function prepareRichTextNode(node, key, content, createNodeId) {
  const str = stringify(content);
  const richTextNode = Object.assign({}, content, {
    id: createNodeId(`${node.id}${key}RichTextNode`),
    parent: node.id,
    children: [],
    [key]: str,
    internal: {
-     type: _.camelCase(`${node.internal.type} ${key} RichTextNode`),
+     type: makeTypeName(`RichTextField`),
      mediaType: `text/richtext`,
      content: str,
      contentDigest: digest(str)
    }
  });
  node.children = node.children.concat([richTextNode.id]);
  return richTextNode;
}

Creating a new type using (_.camelCase(`${node.internal.type} ${key} RichTextNode`)) seems unnecessary. All rich text nodes share the same fields and field types regardless of parent node.

I’d be happy to submit a PR for either of those two approaches.

When you say 7000 nodes, does that mean 7000 pages? And is it 1000x7 or is that 7000x7?

We only build out 2163 pages in both the 6-locale and 7-locale tests. You can check the breakdown of nodes by type in the files I attached in the issue description. In other words, just adding a new locale in Contentful does not result in more pages in our build but it slows down the earlier parts of gatsby tremendously: the source nodes and build schema stages.

A quick example of what’s happening… Say we have:

• 1000 article entries and articles don’t have any localized fields
• 6 locales enabled in Contentful

In Gatsby, we see 6000 ContentfulArticle created and for all but the default locale, the nodes are empty i.e. the fields are all null

Out of curiosity, could you paste the entire output from console (before / after) for a cold build? gatsby clean; gatsby build

Here’s the full output of running the same project with the existing 6 locales and then enabling a 7th locale in Contentful and rerunning (both runs with cleaned cache).

full-build-6-locales.txt full-build-7-locales.txt

[pieh]

Another idea: what kind of things do you pass as a context when you programatically create pages in gatsby-node?

We found that passing large objects there has detrimental effects to schema building step performance