gatsby-source-wordpress: Build failure - WP URL might accidentally be stripped of subdomain

Hi there,

I’m seeing the following build failure using v.3.3.0 of the plugin (also repro’ed using v.3.0.0 version).

Note: I use this plugin with a wordpress instance hosted at wordpress.leadtoreadkc.org, while the actual Gatsby site is hosted at leadtoreadkc.org. I suspect this fact might be causing the error, given the error message below

evannichols:_dev/lead-to-read ‹master*›$ gatsby develop
success open and validate gatsby-configs - 0.064s
warn Warning: there are unknown plugin options for "gatsby-source-wordpress-experimental":
debug.graphql.copyHtmlResponseOnError
success load plugins - 1.423s
success onPreInit - 0.046s
success initialize cache - 0.091s
success copy gatsby files - 0.120s
success onPreBootstrap - 0.028s
success  gatsby-source-wordpress  ensuring plugin requiremenst are met - 5.073s
⠀
info  gatsby-source-wordpress

	This is either your first build or the cache was cleared.
	Please wait while your WordPress data is synced to your Gatsby cache.

	Maybe now's a good time to get up and stretch? :D

success  gatsby-source-wordpress  ingest WPGraphQL schema - 2.438s
success createSchemaCustomization - 7.570s
success  gatsby-source-wordpress  fetch root fields - 2.494s
success  gatsby-source-wordpress  ContentType - 3.299s - fetched 3
success  gatsby-source-wordpress  Comment - 6.783s - fetched 16
success  gatsby-source-wordpress  Category - 7.428s - fetched 13
success  gatsby-source-wordpress  Menu - 1.806s - fetched 1
success  gatsby-source-wordpress  Page - 14.492s - fetched 29
success  gatsby-source-wordpress  PostFormat - 2.650s - fetched 0
success  gatsby-source-wordpress  Tag - 10.174s - fetched 55
success  gatsby-source-wordpress  Post - 12.994s - fetched 59
success  gatsby-source-wordpress  Taxonomy - 2.348s - fetched 3
success  gatsby-source-wordpress  UserRole - 2.402s - fetched 0
success  gatsby-source-wordpress  User - 3.562s - fetched 4

 ERROR

HTTPError: Response code 404 (Not Found)
    at Request._onResponseBase (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/got/dist/source/core/index.js:896:31)
    at Request._onResponse (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/got/dist/source/core/index.js:931:24)
    at ClientRequest.<anonymous> (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/got/dist/source/core/index.js:945:23)
    at Object.onceWrapper (events.js:299:26)
    at ClientRequest.emit (events.js:214:15)
    at ClientRequest.origin.emit (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/@szmarczak/http-timer/dist/source/index.js:39:20)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:583:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:115:17)
    at TLSSocket.socketOnData (_http_client.js:456:22)
    at TLSSocket.emit (events.js:209:13)
    at addChunk (_stream_readable.js:305:12)
    at readableAddChunk (_stream_readable.js:286:11)
    at TLSSocket.Readable.push (_stream_readable.js:220:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:182:23) {
  name: 'HTTPError',
  code: undefined,
  timings: {
    start: 1606519178323,
    socket: 1606519178323,
    lookup: 1606519178334,
    connect: 1606519178340,
    secureConnect: 1606519178352,
    upload: 1606519178353,
    response: 1606519178370,
    end: 1606519178383,
    error: undefined,
    abort: undefined,
    phases: {
      wait: 0,
      dns: 11,
      tcp: 6,
      tls: 12,
      request: 1,
      firstByte: 17,
      download: 13,
      total: 60
    }
  }
}

// (from evan) - IMPORTANT! Why is it trying to download image from leadtoreadkc.org
// instead of wordpress.leadtoreadkc.org?
warn Received the below error when trying to fetch
https://www.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg
from Post #2709 "Readers on the Rooftop Happy Hour"

https://wordpress.leadtoreadkc.org/wp-admin/post.php?post=2709&action=edit



 ERROR

 gatsby-source-wordpress  undefined

not finished source and transform nodes - 96.990s
not finished  gatsby-source-wordpress  fetching nodes - 96.847s - 239 total
not finished  gatsby-source-wordpress  creating nodes - 58.324s
not finished  gatsby-source-wordpress  MediaItem - 58.333s - fetched 56
not finished Downloading remote files - 55.274s
not finished Generating image thumbnails - 54.957s

• It’s failing with a NOT_FOUND when trying to fetch https://www.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg, (it’s missing the subdomain).
• https://wordpress.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg does indeed exist, but trying to visit https://www.wordpress.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg in the browser hangs… I think due to the www prefix.

Is the plugin stripping out the subdomain for some reason? Or is this a configuration issue on my part?

Please let me know if I can provide any more debugging info.

I have shared a branch of the affected repo that reproduces this issue with you (https://github.com/etnichols/lead-to-read/tree/wp-exp-bug).

[etnichols]

(warning: incredibly hacky workaround)

I edited this section of process-node.js to check for the incorrect www subdomain and replace it with wordpress:

for (const {
    cheerioImg,
    match
  } of cheerioImages) {
    let htmlImgSrc = ensureSrcHasHostname({
      src: cheerioImg.attribs.src,
      wpUrl
    });

    if(htmlImgSrc.includes('www')){
      helpers.reporter.warn(`\nHTML img src includes www subdomain (incorrect): ${htmlImgSrc}. Fixing...`);
      htmlImgSrc = htmlImgSrc.replace('www', 'wordpress')
    }

    let imageNode = pickNodeBySourceUrlOrCheerioImg({
      url: htmlImgSrc,
      cheerioImg,
      mediaItemNodes
    });

    if (!imageNode && htmlImgSrc) {
      helpers.reporter.warn(`\nWe didn't get a media item node for this image, we need to fetch it and create a file node for it with no media item node. htmlImgSrc for node: ${htmlImgSrc}`)
      // if we didn't get a media item node for this image,
      // we need to fetch it and create a file node for it with no
      // media item node.
      try {
       // tries to fetch the img resource and now it succeeds
     }
...
}

But this is incredibly brittle 😄 the longer term solution is fixing instantiation of wpUrl.

[etnichols]

Thanks for the reply and sounds good to me!