Gatsby-source-wordpress fetches random amount but rarely all acf_media

Description

I am trying to build a gatsby site with wordpress using acf fields and gatsby-source-wordpress. Unfortunately since adding a a few posts I run into the issue that not all but different images are being fetched every time. The problem is hard to track as I don’t get any error messages and it appears to be random.

The result is usually even worse when deploying to netlify. The project I am working on currently has only 13 posts and 109 wordpress__acf_media to fetch in total.

I looked into similar issues but non of the solutions worked for me. The same issues occurred in previous node versions and before updating all the modules. work on two other websites where I use wordpress-source-plugin as well and the result is very similar.

-> wordpress__acf_options fetched : 1 -> wordpress__acf_posts fetched : 13 -> wordpress__acf_pages fetched : 1 -> wordpress__acf_media fetched : 93 -> wordpress__acf_blocks fetched : 0 -> wordpress__acf_categories fetched : 5 -> wordpress__acf_tags fetched : 0 -> wordpress__POST fetched : 13 -> wordpress__PAGE fetched : 1 -> wordpress__wp_media fetched : 109 -> wordpress__wp_taxonomies fetched : 1 -> wordpress__CATEGORY fetched : 5

The result looks different every time: success Downloading remote files - 6.985s - 10/109 15.60/s success Downloading remote files - 45.059s - 58/109 2.42/s success Downloading remote files - 18.224s - 39/109 5.98/s

Steps to reproduce

Possible conflict with a wordpress plugin? Although I tried to deactivate and delete the ones that are not related to acf and the result didn’t change.

Wordpress Plugins used:

• ACF to REST Api • Advanced Custom Fields PRO • Category Order and Taxonomy Terms Order • TinyMCE Advanced • Webhook Netlify Deploy • WP REST API CacheDeactivate

gatsby-config.js

{
      resolve: `gatsby-source-wordpress`,
      options: {
        baseUrl: `arc.im-burrow.com`,
        protocol: `https`,
        hostingWPCOM: false,
        useACF: true,
        verboseOutput: false,
        includedRoutes: [
          "**/categories",
          "**/posts",
          "**/pages",
          "**/media",
          "**/tags",
          "**/taxonomies",
        ],
        excludedRoutes: [
          "**/comments",
          "**/users",
        ],
        keepMediaSizes: false,
        concurrentRequests: 10,
      }
    },

I assume it is not a graphql issue, never the less:

query {
    allWordpressPost {
      edges {
        node {
          title
          path
          categories{
            slug
          }
          acf {
            index {
              localFile {
                childImageSharp {
                  fluid(maxWidth: 700){
                    ...GatsbyImageSharpFluid_withWebp
                  }
                }
              }
            }
          }
        }
      }
    }
    allWordpressCategory {
      edges {
        node {
          slug
        }
      }
    }
    
  }

Expected result

All images should be downloaded and image.localFile should be accessible.

Actual result

Random amount of media is not being downloaded and return “null”.

Environment

System: OS: macOS 10.15.2 CPU: (4) x64 Intel® Core™ i5-5257U CPU @ 2.70GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 12.13.0 - /usr/local/bin/node Yarn: 1.10.1 - /usr/local/bin/yarn npm: 6.12.0 - /usr/local/bin/npm Languages: Python: 2.7.16 - /usr/bin/python Browsers: Chrome: 78.0.3904.108 Safari: 13.0.4 npmPackages: gatsby: ^2.18.10 => 2.18.10 gatsby-image: ^2.2.34 => 2.2.34 gatsby-plugin-manifest: ^2.2.31 => 2.2.31 gatsby-plugin-offline: ^3.0.27 => 3.0.27 gatsby-plugin-react-helmet: ^3.1.16 => 3.1.16 gatsby-plugin-sass: ^2.1.24 => 2.1.24 gatsby-plugin-sharp: ^2.3.5 => 2.3.5 gatsby-source-filesystem: ^2.1.40 => 2.1.40 gatsby-source-wordpress: ^3.1.51 => 3.1.51 gatsby-transformer-sharp: ^2.3.7 => 2.3.7 npmGlobalPackages: gatsby-cli: 2.8.16

Any Ideas?

[TylerBarnes]

@BURROO if that fixed your problem it’s an issue with the amount of concurrent requests overloading your server. Local servers can also suffer from this problem, although you could likely tweak your local server until it can handle that many concurrent connections. For ex I’m using Local by flywheel locally and it’s actually far slower and can handle less concurrent connections than the pantheon site I’m using for testing.

I’m currently building out the next version of gatsby-source-wordpress which is a ground-up rewrite. I will be adding a retry with exponential backoff algorithm for fetching images and data, so this shouldn’t happen anymore in the next major version.

[BURROO]

@TylerBarnes thank you for the clarification.

@jromme It took me some time to figure this out as well. I added this snippet to the package.json inside “scripts”:

"build": "GATSBY_CONCURRENT_DOWNLOAD=5 gatsby build",
"develop": "GATSBY_CONCURRENT_DOWNLOAD=5 gatsby develop",

And then instead of gatsby develop or gatsby build, I was running npm run develop or npm run build.

If you also work with netlify for the remote build you can set an environment variable in the settings on your netlify panel.