Running test server fails with JavaScript heap out of memory

Hi,

I have a basic Ember application with a few pages. When I run the test suite, it works the first time but then blows up due to JavaScript heap out of memory.

It’s running in a docker container set up to have 4GB of memory and I set the NODE_OPTIONS="--max-old-space-size=4096" environment variable trying to mitigate the issue.

Nonetheless, once the suite has finished running the memory keeps growing. Even after closing any browser running the tests. It slowly grows without any change being done to any file of the app or any new test run. On my last trial it got killed by OOM-killer before failing.

You can find the report of my last trial before augmenting the memory limit: https://gist.github.com/simonc/36927382e5b8cb54e2cb0269cc19be3e

Let me know how I can help tracking down the issue.

Thanks! 🧡

EDIT: Strangely enough, running the same suite directly on the host (MacOS) doesn’t make the memory blow up.

---

Docker container Output from ember version --verbose && npm --version && yarn --version:

ember-cli: 3.17.0
node: 12.16.3
v8: 7.8.279.23-node.35
uv: 1.34.2
zlib: 1.2.11
brotli: 1.0.7
ares: 1.16.0
modules: 72
nghttp2: 1.40.0
napi: 5
llhttp: 2.0.4
http_parser: 2.9.3
openssl: 1.1.1g
cldr: 36.0
icu: 65.1
tz: 2019c
unicode: 12.1
os: linux x64
6.14.4
1.22.4

Host running MacOS Output from ember version --verbose && npm --version && yarn --version:

ember-cli: 3.17.0
node: 12.14.1
v8: 7.7.299.13-node.16
uv: 1.33.1
zlib: 1.2.11
brotli: 1.0.7
ares: 1.15.0
modules: 72
nghttp2: 1.40.0
napi: 5
llhttp: 2.0.1
http_parser: 2.8.0
openssl: 1.1.1d
cldr: 35.1
icu: 64.2
tz: 2019c
unicode: 12.1
os: darwin x64
6.14.5
1.21.1

[wagenet]

I’ve been seeing this on macOS quite a lot recently. I plan to look into it more soon.

[rwjblue]

Additionally, it should be possible to add your own programmatic heapdump periodically via some quick node scripting. Checkout this script I wrote to grab a few consecutive heapdumps from a FastBoot process:

https://github.com/ember-fastboot/fastboot/blob/master/dev/memory-usage.js

Taking that takeHeapsnapshot function and the basic setup there, you could probably throw this in your ember-cli-build.js and have it do periodic snapshots for you to compare later (depending on how annoying your container setup is, this may be easier):

// tweak to something reasonable based on your timeline to failure
const HEAPSNAPSHOT_INTERVAL = 5000;

const inspector = require('inspector');
const path = require('path');
const fs = require('fs');
const session = new inspector.Session();

session.connect();

let file;
// uses whatever the "current" file is
session.on('HeapProfiler.addHeapSnapshotChunk', m => {
  fs.writeSync(file, m.params.chunk);
});

function takeHeapSnapshot(path) {
  file = fs.openSync(path, 'w');

  return new Promise((resolve, reject) => {
    session.post('HeapProfiler.takeHeapSnapshot', null, (err, r) => {
      fs.closeSync(file);

      if (err) {
        reject(err);
      }

      resolve(r);
    });
  });
}

// make sure the `snapshots` dir exists
fs.mkdirSync('snapshots');

let count = 0;
async function scheduleHeapSnapshot() {
  setTimeout(async () => {
    let filename = `${count++}`.padStart(3, '0');
    await takeHeapSnapshot(`snapshots/${filename}.snapshot`);
    scheduleHeapSnapshot();
  }, HEAPSNAPSHOT_INTERVAL);
}

module.exports = function(defaults) {
  scheduleHeapSnapshot();

  let app = new EmberApp(defaults, {
  });

  return app.toTree();
};

^ was fun to write, hopefully didn’t make too many mistakes writing on my phone 😸