[Bug]: Memory consumption issues on Node JS 16.11.0+

🚨 Using Jest 29, you can use --workerIdleMemoryLimit. See https://jestjs.io/docs/configuration/#workeridlememorylimit-numberstring 🚨

Version

27.0.6

Steps to reproduce

1. Install the latest Node JS (16.11.0 or later) or use the appropriate Docker image
2. Set up a project with a multiplicity Jest tests
3. Run node --expose-gc node_modules/.bin/jest --logHeapUsage and see how the memory consumption starts increasing.

Expected behavior

Since Jest calls global.gc() when Garbage Collector is exposed and --logHeapUsage flag is present, the memory usage should be stable.

Actual behavior

The memory usage increases with every new test

Additional context

We had some issues with Jest workers consuming all available RAM both on CI machine and locally. After doing some research, we found that if we run Jest like the following node --expose-gc node_modules/.bin/jest --logHeapUsage, the heap size remains stable. After upgrading to Node JS v16.11.0, the issue was back. Node v16.10.0 works fine. I believe it was something accidentally introduced in the new Node, but it might be useful to take a look at this from Jest perspective in search of possible workarounds. I’m also having the same behavior on my working machine, environment of which I’m pasting below 👇🏻

Environment

System:
    OS: macOS 11.6
    CPU: (8) x64 Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
  Binaries:
    Node: 16.11.0 - ~/.nvm/versions/node/v16.11.0/bin/node
    Yarn: 1.22.0 - ~/SomeFancyDir/webapp/node_modules/.bin/yarn
    npm: 8.0.0 - ~/.nvm/versions/node/v16.11.0/bin/npm
  npmPackages:
    jest: 27.0.6 => 27.0.6

[amiedes]

We’ve been recently hit by this bug (or combination of them) in several services of our stack. It’s been an inconvenience for use since once the heap size exceeded the 2GB available by GitHub Actions machines, our CI/CD process was failing 100% of the time.

I believe the “modules accumulating in the heap” bug has been present in Jest for some time, but that in combination with the changes in Node from v16.10 to v16.11 has made it much more severe. We’re certain that bug exists in Jest, as we’ve managed to replicate it with a “dummy repo”, as several other people have stated.

The fact that the ticket created in Node JS was closed as a WONTFIX is a bit worrying. I currently lack the knowledge to judge the discussions which happened there, but if that wasn’t a bug at the end probably the solution falls on the Jest side.

After trying to extract as much information from the different GitHub issues that have been created in Jest, TS-Jest and Node, and trying several approaches, the only path for us was to downgrade back to Node v16.10.

These are some of the statistics we gathered in the process:

Node	@jest/core	isolatedModules	runInBand	Comp. cache	Initial H	Final H	H ratio	Time
17.2.0	27.3.1	true	true	true	133 MB	2236 MB	~33 MB	144 s
16.13.0	27.3.1	true	true	true	131 MB	2220 MB	~30 MB	130 s
16.13.0	27.3.1	true	true	false	126 MB	1712 MB	~25 MB	113 s
16.13.0	27.3.1	false	true	true	287 MB	2393 MB	~30 MB	140 s
16.10.0	27.3.1	true	true	true	123 MB	790 MB	~10 MB	77 s
16.10.0	27.3.1	true	true	false	118 MB	1676 MB	~25 MB	110 s
16.10.0	27.3.1	false	true	true	280 MB	952 MB	~10 MB	90 s

Some explanations about the columns:

• Node: version of NodeJS
• @jest/core: version of the Jest core
• isolatedModules: the setting defined via jest.config.json
• runInBand: specified by the --runInBand Jest flag
• Comp. cache: specified by the NodeJS --no-compilation-cache flag
• Initial H: heap size after running the first test
• Final H: heap size after running the last test
• H ratio: heap size increase after each test
• Time: time the whole suite needed to run

The most relevant tickets related to this matter:

• https://github.com/facebook/jest/issues/10550
• https://github.com/facebook/jest/issues/11956
• https://github.com/facebook/jest/issues/12142
• https://github.com/facebook/jest/issues/7311
• https://github.com/facebook/jest/issues/7874
• https://github.com/kulshekhar/ts-jest/issues/1967
• https://github.com/nodejs/node/issues/40014
• https://bugs.chromium.org/p/v8/issues/detail?id=12198

We hope this issue is given the importance it deserves, as keeping the Node version pinned to v16.10 or relying on bigger CI machines to compensate this is a poor strategy for the future.

[a88zach]

Our company created a custom runtime based on the patch above that is located here: https://github.com/reside-eng/jest-runtime

This runtime will only work with Jest v28 and it will not work if you are using ESM modules. There is some additional information in the README.md about using swc if you are also using Typescript, but this is not required to use the custom runtime.

$ yarn install -D @side/jest-runtime

then add the following to your jest.config.js

module.exports = {
  ....
  runtime: '@side/jest-runtime',
};