[Webpack 5]: cache leak with incremental compilation

Bug report

What is the current behavior? We recently upgraded from webpack4 to webpack5 and are experiencing memory leaks when using webpack5 caching.

We have the following caching policy defined:

  "cache": {
    "cacheDirectory": "/home/admin/src/coda/.webpack-cache",
    "type": "filesystem",
    "store": "pack",
    "name": "baseApp",
    "idleTimeoutForInitialStore": 0
  }

When we launch webpack devserver the VM usage climbs to about 2.5GB and then stabilizes. The bundle outputs in about 70s. When we mutate a source file, this correctly triggers a recompile, and memory doubles before hitting our max cap of 6GB. The process shortly OOM’s after that.

We’ve tried various options for the filesystem cache including maxAge, maxGeneration, etc. all with no difference. I also tried removing contentHashs and setting output: {clean: true} to no avail. The only thing that does help is disabling caching completely (cache=false) but that causes a horrible regression in incremental compilation times.

Here’s a snapshot of our running webpack5 config:

{
  "context": "/home/admin/src/coda",
  "devServer": {
    "stats": "errors-only"
  },
  "watchOptions": {
    "ignored": [
      "/home/admin/src/coda/home"
    ]
  },
  "devtool": "cheap-module-source-map",
  "output": {
    "crossOriginLoading": "anonymous",
    "devtoolModuleFilenameTemplate": "[resource-path]",
    "devtoolFallbackModuleFilenameTemplate": "[resource-path]?[contenthash]",
    "path": "/home/admin/src/coda/artifacts/browser-app",
    "filename": "[name].[chunkhash].entry.js",
    "chunkFilename": "[name].[contenthash].chunk.js",
    "publicPath": "/dev/cdn/assets/"
  },
  "resolve": {
    "symlinks": false,
    "fallback": {
      "path": "path-browserify",
      "child_process": false,
      "tls": false,
      "fs": false,
      "net": false,
      "http": "stream-http",
      "https": "https-browserify",
      "stream": "stream-browserify",
      "crypto": "crypto-browserify",
      "os": "os-browserify/browser",
      "zlib": "browserify-zlib"
    },
    "alias": {
      "webpack-intermediate-assets": "/home/admin/src/coda/build/webpack-intermediate-assets",
      "@kr-modules": "/home/admin/src/coda/modules"
    },
    "extensions": [
      ".ts",
      ".tsx",
      ".js",
      ".json"
    ],
    "modules": [
      "/home/admin/src/coda/node_modules"
    ],
    "unsafeCache": true
  },
  "stats": {
    "modules": false,
    "children": false
  },
  "cache": {
    "cacheDirectory": "/home/admin/src/coda/.webpack-cache",
    "type": "filesystem",
    "store": "pack",
    "name": "baseApp",
    "idleTimeoutForInitialStore": 0
  },
  "entry": {
    "browser": "@kr-modules/browser/app/entrypoint",
    "external-form": "@kr-modules/browser/external-form/entrypoint"
  },
  "mode": "development",
  "module": {
    "rules": [
      {
        "exclude": [
          {}
        ],
        "test": {},
        "use": [
          {
            "loader": "cache-loader",
            "options": {
              "cacheDirectory": "/home/admin/src/coda/.cache-loader/baseApp"
            }
          },
          {
            "loader": "thread-loader",
            "options": {
              "poolTimeout": null,
              "workers": 3,
              "name": "ts-pool"
            }
          },
          {
            "loader": "ts-loader",
            "options": {
              "configFile": "tsconfig.dev.json",
              "transpileOnly": true,
              "happyPackMode": true,
              "onlyCompileBundledFiles": true,
              "experimentalFileCaching": true,
              "compilerOptions": {
                "module": "esnext"
              }
            }
          }
        ]
      },
      {
        "test": {},
        "exclude": [
          {}
        ],
        "use": [
          "/home/admin/src/coda/node_modules/mini-css-extract-plugin/dist/loader.js",
          {
            "loader": "css-loader",
            "options": {
              "modules": {
                "localIdentName": "[name]--[local]--[hash:base64:8]",
                "mode": "global"
              },
              "sourceMap": true,
              "url": false,
              "esModule": true
            }
          },
          {
            "loader": "postcss-loader",
            "options": {
              "sourceMap": true
            }
          },
          {
            "loader": "less-loader",
            "options": {
              "sourceMap": true,
              "lessOptions": {
                "modifyVars": {
                  "asset-url-prefix": "'/dev/cdn/assets/795f0f96cc72'",
                  "root-asset-url-prefix": "'/dev/cdn'"
                }
              }
            }
          }
        ]
      },
      {
        "test": {},
        "use": {
          "loader": "file-loader",
          "options": {
            "name": "img/[name].[contenthash].[ext]"
          }
        }
      },
      {
        "test": {},
        "loader": "null-loader"
      }
    ]
  },
  "plugins": [
    {
      "resourceRegExp": {},
      "newContentRegExp": {}
    },
    {
      "definitions": {
        "process.env.NODE_ENV": "\"development\"",
        "process.env.LOAD_DOCUMENTATION": "false"
      }
    },
    {},
    {
      "paths": [
        {},
        {}
      ]
    },
    {
      "options": {
        "basePath": "",
        "fileName": "../server/baseApp.manifest.json",
        "filter": null,
        "map": null,
        "publicPath": null,
        "removeKeyHash": {},
        "sort": null,
        "transformExtensions": {},
        "useEntryKeys": false,
        "writeToFileEmit": false
      }
    },
    {
      "options": {
        "filename": "[name].[contenthash].css",
        "ignoreOrder": false,
        "chunkFilename": "[name].[contenthash].chunk.css"
      }
    },
    {
      "options": {
        "filter": {},
        "allow": "(Apache-2.0 OR BSD-2-Clause OR BSD-3-Clause OR ISC OR MIT OR OFL-1.1 OR Unlicense OR WTFPL OR W3C OR Zlib)",
        "ignore": [
          "highcharts@7.2.2",
          "coda-packs-sdk@0.0.1",
          "coda-icons@0.0.1"
        ],
        "override": {
          "@improbable-eng/grpc-web@0.14.0": {
            "licenseName": "Unlicense"
          }
          // shortened...
        },
        "emitError": true,
        "outputFilename": "OpenSourceSoftwareLicenses-baseApp.json"
      }
    },
    {
      "options": {},
      "timeEventData": {},
      "smpPluginAdded": false
    },
    {
      "definitions": {}
    },
    {
      "definitions": {
        "Buffer": [
          "buffer",
          "Buffer"
        ],
        "process": "process/browser"
      }
    },
    {
      "options": {
        "resourceRegExp": {}
      }
    }
  ],
  "optimization": {
    "minimize": false,
    "concatenateModules": false,
    "minimizer": [
      // TerserPlugin
      {
        "options": {
          "test": {},
          "extractComments": true,
          "cache": true,
          "parallel": 8,
          "terserOptions": {
            "ecma": 2018,
            "compress": {
              "ecma": 2018,
              "keep_classnames": true,
              "unsafe": true,
              "unsafe_arrows": false,
              "unsafe_math": false
            }
          }
        }
      },
      // OptimizeCSSAssetsPlugin
      {
        "pluginDescriptor": {
          "name": "OptimizeCssAssetsWebpackPlugin"
        },
        "options": {
          "assetProcessors": [
            {
              "phase": "compilation.optimize-chunk-assets",
              "regExp": {}
            }
          ],
          "assetNameRegExp": {},
          "cssProcessorOptions": {
            "map": {
              "annotationPrefix": "",
              "inline": false
            }
          },
          "cssProcessorPluginOptions": {}
        },
        "phaseAssetProcessors": {
          "compilation.optimize-chunk-assets": [
            {
              "phase": "compilation.optimize-chunk-assets",
              "regExp": {}
            }
          ],
          "compilation.optimize-assets": [],
          "emit": []
        },
        "deleteAssetsMap": {}
      }
    ]
  }
}

Here’s sample output during the compile:

2021-04-13-17:59:20 0|app-webpack     | [baseApp] Change in modules/global-style/colors.less; rebuilding…
2021-04-13-18:00:33 0|app-webpack     |  SMP  ⏱
2021-04-13-18:00:33 0|app-webpack     | General output time took 1 min, 12.86 secs
2021-04-13-18:00:33 0|app-webpack     |  SMP  ⏱  Loaders
2021-04-13-18:00:33 0|app-webpack     | mini-css-extract-plugin, and
2021-04-13-18:00:33 0|app-webpack     | css-loader, and
2021-04-13-18:00:33 0|app-webpack     | postcss-loader, and
2021-04-13-18:00:33 0|app-webpack     | less-loader took 1 min, 5.029 secs
2021-04-13-18:00:33 0|app-webpack     |   module count = 750
2021-04-13-18:00:33 0|app-webpack     | css-loader, and
2021-04-13-18:00:33 0|app-webpack     | postcss-loader, and
2021-04-13-18:00:33 0|app-webpack     | less-loader took 1 min, 4.92 secs
2021-04-13-18:00:33 0|app-webpack     |   module count = 750
2021-04-13-18:00:33 0|app-webpack     | assets by status 10.8 MiB [cached] 6 assets
2021-04-13-18:00:33 0|app-webpack     | assets by status 43.7 MiB [emitted]
2021-04-13-18:00:33 0|app-webpack     |   assets by info 43.5 MiB [immutable]
2021-04-13-18:00:33 0|app-webpack     |     assets by path *.js 41.8 MiB
2021-04-13-18:00:33 0|app-webpack     |       asset browser.6762f8d648ff98edcf15.entry.js 23.1 MiB [emitted] [immutable] (name: browser) 1 related asset
2021-04-13-18:00:33 0|app-webpack     |       asset external-form.61557166b57a93b7fb7f.entry.js 18.7 MiB [emitted] [immutable] (name: external-form) 1 related asset
2021-04-13-18:00:33 0|app-webpack     |     assets by path *.css 1.65 MiB
2021-04-13-18:00:33 0|app-webpack     |       asset browser.e5ff57d15328ceb4a662.css 1000 KiB [emitted] [immutable] (name: browser) 1 related asset
2021-04-13-18:00:33 0|app-webpack     |       asset external-form.31948c4ab1b96ab265b5.css 689 KiB [emitted] [immutable] (name: external-form) 1 related asset
2021-04-13-18:00:33 0|app-webpack     |   asset OpenSourceSoftwareLicenses-baseApp.json 215 KiB [emitted]
2021-04-13-18:00:33 0|app-webpack     |   asset ../server/baseApp.manifest.json 2.79 KiB [emitted]
2021-04-13-18:00:33 0|app-webpack     | Entrypoint browser 24.1 MiB (20.5 MiB) = browser.e5ff57d15328ceb4a662.css 1000 KiB browser.6762f8d648ff98edcf15.entry.js 23.1 MiB 2 auxiliary assets
2021-04-13-18:00:33 0|app-webpack     | Entrypoint external-form 19.4 MiB (17.1 MiB) = external-form.31948c4ab1b96ab265b5.css 689 KiB external-form.61557166b57a93b7fb7f.entry.js 18.7 MiB 2 auxiliary assets
2021-04-13-18:00:33 0|app-webpack     | webpack 5.32.0 compiled successfully in 72866 ms

Versioning information: webpack: 5.32.0 webpack-cli: 4.5.0 webpack-dev-server: ^.11.2

I noticed similar issues reported in https://github.com/webpack/webpack/issues/12947. The proposal there was to use a memory cache and set output.clean. I’ve tried that with {type: "memory", maxGenerations: 1} and output.clean, as well as stripping contentHashes. I still see the issue.

Anything else I can do to help narrow down the underlying issue? Any workaround that would maintain caching without the OOM?

Thanks, Nigel.

If the current behavior is a bug, please provide the steps to reproduce. I’m just running:

webpack-cli -w --color --env dev--config=...

and then mutate a source file.

What is the expected behavior? The memory working-set should be stable and not grow without bounds.

Other relevant information: webpack version: 5.32.0 Node.js version: 14.16.1 Operating System: MacOS (or Linux Buster) Additional tools:

[alexander-akait]

@kanoshin Can you test again, please update webpack https://github.com/webpack/webpack/releases/tag/v5.42.0 and less-loader https://github.com/webpack-contrib/less-loader/releases/tag/v10.0.1

[kanoshin]

@alexander-akait attaching the snapshots. It’s almost impossible to take the snapshot in our larger app, so I decided to use a smaller one but it still shows a very disproportional memory usage comparing to the codebase size. This app is around 223MB of source code with node_modules but initial memory usage is 3x of that amount and then it keeps growing between rebuilds.

What I’m trying to say is that it’s not just a memory leak I’m reporting but most importantly is that simply the initial memory consumption for larger apps makes filesystem cache unusable for larger codebases.

Google drive with zip archive: https://drive.google.com/file/d/1_XR27NpTQnYcOnFZFowqJqcB1FEbe1g0/view?usp=sharing.