Delta builds for very large sites

So, as a POC I have successfully built a static site that has ~100k posts using the built in react-static build which is awesome! Takes about ~45 minutes on my personal, run of the mill MacBook. For the record, out of the box both Gatsby and Next.js would blow up with memory issues before the 10k mark so I am super pumped.

At this point I am doing some research into the best way to be able to update any number of posts and only re-build the delta (or the newly published content). This post data would come from an API call in the static.config.js file, probably something like what exists in the examples. Can anyone offer any insight into the best way of building only the new or updated posts, and leaving the rest of the existing site as is? I was thinking I would have to write a custom build / webpack script for this, but wanted to leverage any existing hooks / infrastructure. I am still looking at other static-site-generators for inspiration, but I figured since someone else will inevitably have a similar use case I wanted to ask here.

Thanks!

Matthew

[tannerlinsley]

That’s great to hear! This is a really cool idea, too. As far as only updating specific posts, that would probably need some tweaking in the architecture. But it could be done. Basically, it would involve a few things:

• Keeping the dist folder around in your project
• Not cleaning the dist folder on build
• A special update CLI command or option in the config
• A special getRoutesUpdate callback or something similar to fetch the data for the updated posts and return which routes should be updated.

Some other thoughts:

• The bundle would probably need to be generated again

[tannerlinsley]

After a lot of discussion on this topic both internally and externally, I think we’ve decided to take the direction of the library towards “fast immutable” builds. Delta builds introduce a few things that are difficult to prepare for:

• Data dependencies and relationships. Given RS’s data agnostic approach, this would be almost impossible.
• Server or git related caching mechanisms. This is not RS’s arena or responsibility.

We feel our efforts will be much better invested into producing faster builds for the library.