Improve Persistent Caching for multiple configurations

Feature request

First, I don’t know if all this even makes sense, just wanted to expose my use-case and get some feedback.

I work on the Docusaurus framework, we are migrating to Webpack 5 and are happy to leverage the benefits of the new persistent cache.

This is not a critical feature I absolutely need, I am just wondering if, in addition to the great benefits of this new caching, there’s still room for even better improvements.

I thought about asking this to @alexander-akait in our PR (as he proposed to help), but I thought it would be better to discuss this in a dedicated issue.

What is the expected behavior?

Improve incremental build performance when you build multiple Webpack configurations in a row, all of them being quite similar and using some shared code.

What is motivation or use case for adding/changing the behavior?

The Docusaurus build system is using a client (browser) config and a server (SSR) config.

When using the i18n feature, we loop over a list of locales and create one distinct SPA per locale (we’ll likely explore module federation later).

The process looks a bit like that:

function compile(configs: Configuration[]): Promise<void> {
  return new Promise((resolve, reject) => {
    const compiler = webpack(configs);
    compiler.run((err, stats) => {
      //
      compiler.close(errClose => {
        // ...
      });
    });
  });
}

const locales = ["en", "fr", "de"];

// sequential (on purpose)
for (locale of locales) {
  await compile([
    getConfig({ server: true, locale }),
    getConfig({ server: false, locale })
  ]);
}

The build time perfs I see for a single locale is already great:

• cold cache: 100s
• warm cache: 25s

But if I build multiple locales with a cold cache, I end up with 100s * number of locale, so 300s for 3 locales for example. I am wondering if Webpack could not be able to build “en” and then reuse some part of the cache to speed up the build of “fr” and “de”?

We have multiple i18n deployment strategies (subdomain fr.domain.io vs subpath domain.io/fr) By default, each locale has a different output.path (like dist/fr and output.publicPath (like /fr/), but it’s also possible to build the site for the subdomain strategy.

My impression is that the current caching system will bail-out of using the cache if any config has changed between 2 runs (config is provided through CLI args, I’m not editing buildDependencies __filename)

That’s why by default I need to use a distinct cache name per locale to be able to make the incremental build work:

cache: {
   type: 'filesystem',
   name: `${isServer ? "server" : "client"}-${mode}-${locale}`,
}

I’ve tried to use the non-default i18n deployment strategy (which use a subdomain deployment, for which path: 'dist' and publicPath: '/' for all locales), and use a shared cache for all locales: name:${isServer ? “server” : “client”}-${mode}`.

Some tests I’ve done:

clear cache
build --locale en => 100s
build --locale fr => 50s

The second build is significantly faster because the cache created for “en” has been successfully leveraged for “fr”. But “en” and “fr” are not strictly equivalent code (as it diverse on translated texts and mdx docs), so quite expectedly, building “fr” after “en” is not as fast as building twice for the same locale:

clear cache
build --locale en => 100s
build --locale en => 25s

Now let’s try something else:

clear cache
build --locale en => 100s
build --locale fr => 50s
build --locale en => 50s

What we can see is that the last build is not as fast as if we built twice in “en” in a row. This lets me think that the persistent cache only preserves the cache entries of the very last build.

How should this be implemented in your opinion?

Make it possible to keep entries of older builds in the cache? I’d like to have these perfs:

clear cache
build --locale en => 100s
build --locale fr => 50s
build --locale en => 25s (-25s reduction!)

Make it possible to share a cache across multiple builds using different but slightly similar config (ie for different locales, only output.path, output.publicPath and some i18n.json and myDoc.mdx are different, but much of code remains shared between all locales)

Are you willing to work on this yourself?

I don’t feel skilled enough to work on this.

---

Note: I’ve seen the build-performance/#multiple-compilations info, but thought this may be stale doc? Should we still use cache-loader for such usecase? Do we need parallel-webpack when webpack can compile multiple configs at once? (also last release was 1 year ago)

Note: our server/client config are quite different, the goal for our usecase is more to share the cache on the “locales axis” rather than the “client/server axis”.

[alexander-akait]

Each name name: ${isServer ? “server” : “client”}-${mode}-${locale}` create own file for cache.

This lets me think that the persistent cache only preserves the cache entries of the very last build.

Yep.

I think you want to say here about shared cache here between different compilers.

It makes sense. Maybe before we need solve https://github.com/webpack/webpack/issues/10400.

/cc @sokra

[webpack-bot]

Issue was closed because of inactivity.

If you think this is still a valid issue, please file a new issue with additional information.