FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory

Hi!

We currently face a Javascript heap out of memory which seems to be systematically caused by using express-validator according to the logs below.

2020-01-30 13:00:03: <--- Last few GCs --->
2020-01-30 13:00:03:
2020-01-30 13:00:03: [21946:0x3043640] 70389862 ms: Scavenge 891.3 (918.4) -> 877.9 (918.4) MB, 2.5 / 0.0 ms  (average mu = 0.986, current mu = 0.987) allocation failure
2020-01-30 13:00:03: [21946:0x3043640] 70390432 ms: Scavenge 893.5 (919.1) -> 880.1 (919.1) MB, 2.5 / 0.0 ms  (average mu = 0.986, current mu = 0.987) allocation failure
2020-01-30 13:00:03: [21946:0x3043640] 70391005 ms: Scavenge 895.6 (919.6) -> 882.2 (919.6) MB, 2.5 / 0.0 ms  (average mu = 0.986, current mu = 0.987) allocation failure
2020-01-30 13:00:03:
2020-01-30 13:00:03:
2020-01-30 13:00:03: <--- JS stacktrace --->
2020-01-30 13:00:03:
2020-01-30 13:00:03: ==== JS stack trace =========================================
2020-01-30 13:00:03:
2020-01-30 13:00:03:     0: ExitFrame [pc: 0x1374fd9]
2020-01-30 13:00:03: Security context: 0x2120d08c08a1 <JSObject>
2020-01-30 13:00:03:     1: middleware(aka middleware) [0x201b75e15789] [/home/project/node_modules/express-validator/src/middlewares/check.js:~20] [pc=0x2b7a08665fe5](this=0x1564065c04a9 <undefined>,0x39fa0b11b8d9 <IncomingMessage map = 0x1bf090170b81>,0x39fa0b11bfc1 <ServerResponse map = 0x2252fd1a13b1>,0x39fa0b12b399 <JSFunction next (sfi = 0x1c95541b1969)>)
2020-01-30 13:00:03:  ...
2020-01-30 13:00:03:
2020-01-30 13:00:03: FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory

We use express-validator latest version 6.3.1. The code never uses directly the check function (neither does any dependency) but we do use validationResult, param, body, header and query functions from the library.

We already try without success giving --max-old-space-size=8192 as arguments to node and max_memory_restart: '8192M' as argument to pm2. But unfortunately, it only postpone the inevitable.

Maybe this answer on SO can help guide the investigation.

Any help on this will be appreciated! 😃

[gustavohenke]

The good news is we finally reach production last Monday with the update and there is not a single restart since then

Ah that’s so good to hear! Would love to know more and dig into this issue later if it reappears.

Unlike me, thank you for your quick response and action!

If I only could reply this fast to every issue opened here 😓

[skyless]

Hi @gustavohenke,

First, I’m absolutely sorry it’s taken me so long to get back to you. I hope you’re safe and taking care of your family!

Which kind of validations do you have? I’ll give a try at replicating the bug.

As simple as validating all the request params, body, query attributes types and optionality.

Also, can you please let me know how long/how many requests does your server take before dying?

The trafic is variable but I would say I receive 2 000 000 requests a day on the problematic service.

I’m wondering if that’s something that happens at scale, or it’s quite common.

It looks like to me it’s rather at scale, for beginning being harmful. To give you an idea, I had 86 restarts in 50 days frame. Almost 2 restarts per day.

No node application should need this much memory!

Totally agree but it was worth to try. ¯_(ツ)_/¯

The good news is we finally reach production last Monday with the update and there is not a single restart since then! It looks like the issue was indeed resolved by compiling the code to ES2017.

I’ll still keep an eye in the coming days and I’ll not hesitate to keep you posted if the issue appears again.

Unlike me, thank you for your quick response and action!