It takes 8 hours to run Gatsby Develop + Drupal 8 store (40 000 products)

Summary

Hello,

I have website built on Headless Drupal 8 with around 30 000 products (+10 000 out of stock) I started with building frontend on Gatsby without connecting to my Drupal 8 backend when I tried to connect it using gatsby-source-drupal plugin a lot of different problems started to show up.

Each product have 5 images so in total there should be around 200 000 images

Basic example

1. Javascript out of heap error - when reaching 2 GB (solved) by adding new command in package.json “heavyload”: “node --max-old-space-size=14192 ./node_modules/gatsby/dist/bin/gatsby.js develop” and running: npm run heavyload

2. Fetching - info Starting to fetch data from Drupal success Fetch data from Drupal - 2623.108s - take around 43 minutes

3. info Downloading remote files from Drupal <- and now the real problem begins, in order to run the Gatsby Develop our script starts to download all files into cache files (around 50GB) unfortunately if it fails while downloading those (50 GB) files and I want to run Gatsby develop again, I need to download all 50 GB from scratch (it already happened twice to me) and it takes around 8 hours to download 80 gb of files.

4. After running the Gatsby Develop for past 20 hours I was unable to finish this process even once while connected to my Drupal website I’m attaching log of failed attempt

0 info it worked if it ends with ok
1 verbose cli [
1 verbose cli   'C:\\Program Files\\nodejs\\node.exe',
1 verbose cli   'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'heavyload'
1 verbose cli ]
2 info using npm@6.14.4
3 info using node@v12.16.3
4 verbose run-script [ 'preheavyload', 'heavyload', 'postheavyload' ]
5 info lifecycle gatsby-starter-hello-world@0.1.0~preheavyload: gatsby-starter-hello-world@0.1.0
6 info lifecycle gatsby-starter-hello-world@0.1.0~heavyload: gatsby-starter-hello-world@0.1.0
7 verbose lifecycle gatsby-starter-hello-world@0.1.0~heavyload: unsafe-perm in lifecycle true
8 verbose lifecycle gatsby-starter-hello-world@0.1.0~heavyload: PATH: C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;C:\Users\Karol\nauka-gatsby\mywebsite\node_modules\.bin;C:\Python38\Scripts\;C:\Python38\;C:\Python27\;C:\Python27\Scripts;C:\WINDOWS\system32; ##commented out some of things from here ### C:\Users\Karol\AppData\Roaming\npm;C:\Users\Karol\AppData\Local\Programs\Microsoft VS Code
9 verbose lifecycle gatsby-starter-hello-world@0.1.0~heavyload: CWD: C:\Users\Karol\nauka-gatsby\mywebsite
10 silly lifecycle gatsby-starter-hello-world@0.1.0~heavyload: Args: [
10 silly lifecycle   '/d /s /c',
10 silly lifecycle   'node --max-old-space-size=8192 ./node_modules/gatsby/dist/bin/gatsby.js develop'
10 silly lifecycle ]
11 silly lifecycle gatsby-starter-hello-world@0.1.0~heavyload: Returned: code: 1  signal: null
12 info lifecycle gatsby-starter-hello-world@0.1.0~heavyload: Failed to exec heavyload script
13 verbose stack Error: gatsby-starter-hello-world@0.1.0 heavyload: `node --max-old-space-size=8192 ./node_modules/gatsby/dist/bin/gatsby.js develop`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\index.js:332:16)
13 verbose stack     at EventEmitter.emit (events.js:310:20)
13 verbose stack     at ChildProcess.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:310:20)
13 verbose stack     at maybeClose (internal/child_process.js:1021:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
14 verbose pkgid gatsby-starter-hello-world@0.1.0
15 verbose cwd C:\Users\Karol\nauka-gatsby\mywebsite
16 verbose Windows_NT 10.0.18362
17 verbose argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "heavyload"
18 verbose node v12.16.3
19 verbose npm  v6.14.4
20 error code ELIFECYCLE
21 error errno 1
22 error gatsby-starter-hello-world@0.1.0 heavyload: `node --max-old-space-size=8192 ./node_modules/gatsby/dist/bin/gatsby.js develop`
22 error Exit status 1
23 error Failed at the gatsby-starter-hello-world@0.1.0 heavyload script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Motivation

I’m not sure what I’m doing wrong, but I don’t know honestly how to deal with it, I want to connect my Gatsby website with my Drupal 8 website, but if it takes around 8 hours to connect and after 8 hours I get error and have to try from scratch it, I really don’t know what to do.

Perhaps it is my API fault, or Gatsby Drupal plugin have missing configuration, I can share my API but I wouldn’t want to make it public.

[ascorbic]

What I did in a similar situation was to rename file--file to something else with jsonapi_extras, which meant it still came through to Gatsby, but didn’t download anything. On the Drupal side we set up s3fs for the images, and then linked the bucket to ImageKit. I then wrote a custom resolver to generate gastby-image compatible types, using ImageKit URLs generated from the s3 URLs coming from Drupal.

[smthomas]

It looks like we have a few problems here. Downloading all of the images is taking a long time. The gatsby-source-drupal plugin is going to download these images as long as they are referenced in the JSON provided by JSON:API.

The issue you linked is one potential solution to help with that. I will follow up on that issue to see what is blocking it from getting merged but it appears it does require some additional work.

It looks like the commerce_api module will not work with jsonapi_extras. This is going to cause other problems down the line because the jsonapi_extras module is required if you want things like live preview or incremental builds to work. It might be worth following up in this thread to see what is causing to conflicts and if it is something commerce_api will be fixing in the future - https://www.drupal.org/project/commerce_api/issues/3121480

The idea of third party image hosting is a good one. There are options such as Cloudinary or you could host them on your Drupal site and potentially use the consumer_image_styles plugin. Other third party hosted CMS’s such as Contentful and DatoCMS use this approach (which is why their build times are much faster). The images are never pulled down locally and the correct image is displayed at runtime based on the browser size. This would definitely speed up your builds but we would still need to figure out how to make sure the files were not downloaded (which is why the jsonapi_extras module would be helpful).