Slow "update schema" part of the build process

Description

The “update schema” part of the build process is very slow - close to 20 minutes for only 850 pages. From what I’ve read it seems that the build process should be much faster than this and considering you’re currently looking for projects with 50k+ pages which would take close to 20 hours at this pace.

Every part of the build process seems reasonably fast, only the “update schema” takes very long and “run queries” also being slower than I would expect.

Steps to reproduce

Github repo https://github.com/MartinMikusat/gatsby-reproduce

I’ve made it create only 50 pages so it doesn’t do more work than necessary to make the point and added some instructions in README for making it create all ~850 pages if you want to test it like that as well. It takes too long even with 50 pages.

Just run npm install and gatsby develop

Expected result

Running the initial build should be fast

Actual result

Running the initial build is slow

Environment

System: OS: Windows 10 10.0.18362 CPU: (4) x64 Intel® Core™ i5-7200U CPU @ 2.50GHz Binaries: Node: 12.14.0 - C:\Program Files\nodejs\node.EXE npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD Languages: Python: 2.7.17 Browsers: Edge: 44.18362.449.0 npmPackages: gatsby: ^2.18.12 => 2.18.12 gatsby-image: ^2.2.38 => 2.2.38 gatsby-plugin-eslint: ^2.0.8 => 2.0.8 gatsby-plugin-material-ui: ^2.1.6 => 2.1.6 gatsby-plugin-postcss: ^2.0.7 => 2.1.19 gatsby-source-sanity: ^5.0.5 => 5.0.5

[vladar]

@MartinMikusat thanks for bringing this up!

It looks like you put too much data into the page context. You put your whole data set into it: https://github.com/MartinMikusat/gatsby-reproduce/blob/master/gatsby-node.esm.js#L83

Page context is not meant for passing data to pages. It is meant to pass simple parameters that could be later passed to the GraphQL query of the page as variables.

There is an overhead associated with context - Gatsby has to visit every field of every object in context to infer SitePageContext type.

You have pages for 850 nodes (SanityLens in your example) and each page context has all the nodes in it. Also, each node has about 30 fields. It means that Gatsby has to visit 850 * 850 * 30 fields (which is approx. 21,7 million).

We could do better by showing a warning in cases like this (when the context is not used appropriately) and we have an issue exactly for this problem: #14213

Sorry if it is a bit confusing and thanks for using Gatsby 💜

[MartinMikusat]

Thanks for the reply. I figured out something similar myself at the end but I have since redesigned the whole website completely, so unfortunately I can’t show a working example of this for others. I guess I can close this issue now. Thanks again!