Node crashes with OOM if you pass pretty-format a large object

🐛 Bug Report

Node runs out of heap space and aborts with stack trace from V8 if you pass a large enough object to pretty-format, for example via toMatch[Inline]Snapshot().

I suppose this ties in with #4645, although that was closed by forcing pretty-format to not serialise the object type in question as a regular object.

To Reproduce

Assuming you run in the default jest-environment-jsdom, document is all you need to provoke this behaviour:

test('break', () => {
  expect(document).toMatchInlineSnapshot('');
});

Expected behavior

I think pretty-format should fail the test with an informative message instead of letting the Jest/Node process die if the serialisation is about to fill the heap.

One way to do this could be to abort object serialisation when when the test would otherwise time out. It takes (much) longer than the default test timeout of 5 seconds to run out of memory on my late 2013 MacBook Pro:

➜  jest-expect-document git:(master) time y jest
yarn run v1.12.1
$ /Users/theneva/code/jest-expect-document/node_modules/.bin/jest

 RUNS  ./index.test.js

<--- Last few GCs --->

[redacted]

<--- JS stacktrace --->

[redacted]

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
yarn jest  47.43s user 1.56s system 185% cpu 26.362 total

Link to repl or repo (highly encouraged)

See screenshot and repro using that test here: https://github.com/theneva/jest-expect-document

Run npx envinfo --preset jest

Paste the results here:

System:
  OS: macOS Sierra 10.12.6
  CPU: (8) x64 Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Binaries:
  Node: 10.6.0 - ~/.nvm/versions/node/v10.6.0/bin/node
  Yarn: 1.12.1 - /usr/local/bin/yarn
  npm: 6.1.0 - ~/.nvm/versions/node/v10.6.0/bin/npm
npmPackages:
  jest: ^23.6.0 => 23.6.0

[SimenB]

I think it makes sense for every utility that comes out of the box with Jest (such as matchers) to honour the test timeouts if possible.

prettyFormat is synchronous - no way to interrupt it in JS.

---

We’ve shipped a new diffing algorithm in Jest 24, so I think we can close this. Might revisit some upper bound on diff size if it’s still an issue

[SimenB]

“sufficiently large object” is tracked in #1772. However, document should be special-cased like other jsdom objects to look like markup after serialization, imo.

@rickhanlonii @pedrottimark agree?