[`gatsby-plugin-mdx`] Stale scope caches causing build failures on valid input

Description

Currently, gatsby-plugin-mdx saves the import scope of MDX nodes to the mdx-scopes-dir cache directory, hashed on their contents (gatsby/on-create-node). At runtime, these scopes are all loaded by loaders/mdx-scopes, merged into a single object, and re-exported.

Of note, these scope files are never invalidated/removed. As such, if any file imported from an .mdx file is renamed, the build will continuously fail until the cache is cleared - even if the imports to the renamed file are amended correctly.

While running gatsby clean does essentially resolve this, it’s an extremely blunt fix, requiring a time consuming rebuild afterwards.

In addition, as a side effect of merging the contents of all scope files in the loader, local names of imports within MDX files must be unique project-wide, as the Object.assign call will kludge any duplicates.

I did take an attempt at a fix, however I’m a little tight on time, and don’t know my way around gatsby’s internals well enough to get a solid fix together. My best guess at an approach was to associate nodes with the mdx scopes they generate, and move the wrapRootElement down to wrapPageElement, where the associated scope file could be imported - theoretically, that would resolve the local name kludging, and avoid stale files from being loaded. I’m not sure how feasible that is, however.

Steps to reproduce

No reproduction repo, it’s possible to reproduce quickly from a starter.

1. npm init gatsby, pick the MDX option in wizard
2. start the development server
3. add an .mdx file to src/pages/, import a file from it
4. rename the imported file, and fix import

Expected result

Build failures are resolved once import is corrected with the new import path.

Actual result

Build continues to fail until stale scope file is removed, typically when the cache is totally wiped.

Environment

  System:
    OS: macOS 10.15.7
    CPU: (12) x64 Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 12.19.1 - /var/folders/yc/bsv5xpmx42l2g2vcn0f3nd7r0000gn/T/yarn--1611198685083-0.2823682586357512/node
    Yarn: 1.22.10 - /var/folders/yc/bsv5xpmx42l2g2vcn0f3nd7r0000gn/T/yarn--1611198685083-0.2823682586357512/yarn
    npm: 6.14.8 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 87.0.4280.141
    Firefox: 76.0.1
    Safari: 14.0.2
  npmPackages:
    gatsby: ^2.26.1 => 2.31.0 
    gatsby-plugin-mdx: ^1.9.0 => 1.9.0 
    gatsby-source-filesystem: ^2.10.0 => 2.10.0 

[LekoArts]

Hi!

We merged https://github.com/gatsbyjs/gatsby/pull/35650 which means that a new major version of gatsby-plugin-mdx will be released on August 16. You can already try it out by installing gatsby-plugin-mdx@next & gatsby@next.

With this new version the above issue is no longer there since the plugin has been rewritten from scratch and no longer uses the mdx-scopes-dir. Since the official MDX webpack loader is used, there’s no longer bundle bloat and every MDX file has it’s own chunk.

[ackwell]

Ah, thanks for the link - hadn’t seen the discussion. Will keep an eye out - may have some time in future to look into it further, but have some deadlines occupying time in the short term.