Be less aggressive at deleting sourced data

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

Gatsby syncs data from local and remote sources into LMDB. We then run queries against our local DB. This syncing takes time and is a frequent cause of slow builds or slow dev server startups as some sources can be quite slow to sync from e.g. WordPress where 10m+ sourcing times aren’t uncommon.

This wouldn’t be a big deal except for the rare full sync when setting up a new site. Except unfortunately Gatsby’s pretty aggressive about deleting the local db of synced data.

Examples of user actions which unexpectedly trigger the local db being deleted:

• adding or upgrading any npm package
• editing gatsby-config.js in any way
• editing gatsby-node.js in any way including common tweaks like editing createPages code

Previous attempts at fixing this:

• https://github.com/gatsbyjs/gatsby/pull/8379
• https://github.com/gatsbyjs/gatsby/pull/32318

Reproduction Link

https://www.gatsbyjs.com/starters/gatsbyjs/gatsby-starter-wordpress-blog

Steps to Reproduce

1. setup a site with a source plugin that supports delta fetches like gatsby-source-wordpress and run gatsby develop
2. restart gatsby develop and observe the sourcing time is much faster as we only sync updated content this time
3. make a trivial edit to gatsby-config.js and restart the dev server
4. observe that the local db is deleted and that gatsby-source-wordpress must refetch all data

Expected Result

gatsby-source-wordpress only refetches all data if I a) upgrade the plugin or b) edit its config.

Actual Result

Many changes I make to a site trigger refetching all data.

Environment

n/a

Config Flags

n/a

[KyleAMathews]

Fixing bugs is always appreciated and for new features, most of the energy will be directed at the active RFCs so testing canaries and helping flesh out the new features is very helpful to ensure robust, fast releases.

[github-actions[bot]]

Hey again!

It’s been 60 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it. Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to comment on this issue or create a new one if you need anything else. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜