Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory leak on NodeJS v12

See original GitHub issue

Our application is instrumenting several custom metrics while processing millions of transactions per hour.

hot-shots v6.8.5 continuously runs out of heap on several of our worker processes when pushing metrics to the Datadog Agent on Ubuntu 18.04, NodeJS 12.14.1.

Herewith the latest crash report.

<--- Last few GCs --->

[13503:0x36718f0] 128653900 ms: Mark-sweep 2005.7 (2084.4) -> 2005.6 (2084.4) MB, 1193.0 / 16.6 ms  (average mu = 0.211, current mu = 0.150) allocation failure GC in old space requested
[13503:0x36718f0] 128655277 ms: Mark-sweep 2005.6 (2084.4) -> 2005.6 (2053.4) MB, 1190.8 / 21.9 ms  (average mu = 0.176, current mu = 0.135) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1376259]
Security context: 0x27803ed408a1 <JSObject>
    1: overrideTags [0x26a9446fd2a1] [/opt/app/node_modules/hot-shots/lib/helpers.js:~30] [pc=0xc92ac44278e](this=0x26a9446fb609 <Object map = 0x123ff6ab29d1>,0x362f61553ac9 <JSArray[5]>,0x196845a3a3a1 <JSArray[2]>,0x0a0be40806e1 <false>)
    2: increment [0xfcc620075e9] [/opt/app/node_modules/hot-shots/lib/statsFunctions.js:~101] [pc=0xc92ac5...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x9dab70 node::Abort() [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 2: 0x9dbd26 node::OnFatalError(char const*, char const*) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 3: 0xb3b18e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 4: 0xb3b509 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 5: 0xce68a5  [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 6: 0xce6f36 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 7: 0xcf2dca v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 8: 0xcf3cd5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
 9: 0xcf5469 v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
10: 0xcf673f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
11: 0xcc2b7b v8::internal::Factory::NewTransitionArray(int, int) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
12: 0xf03877 v8::internal::TransitionsAccessor::Insert(v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
13: 0xeb0ed7 v8::internal::Map::ConnectTransition(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::SimpleTransitionFlag) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
14: 0xeb358e v8::internal::Map::CopyReplaceDescriptors(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>, v8::internal::TransitionFlag, v8::internal::MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
15: 0xeb3b9a v8::internal::Map::CopyAddDescriptor(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
16: 0xeb3dd9 v8::internal::Map::CopyWithField(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
17: 0xeb55a2 v8::internal::Map::TransitionToDataProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::StoreOrigin) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
18: 0xea5c7f v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::StoreOrigin) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
19: 0xeda0d6 v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
20: 0xedacc7 v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
21: 0x100af54 v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
22: 0x100c67a v8::internal::Runtime_SetKeyedProperty(int, unsigned long*, v8::internal::Isolate*) [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]
23: 0x1376259  [/home/ubuntu/.nvm/versions/node/v12.14.1/bin/node]

What could cause this behaviour and are there any suggestions for fine tuning the API ?

Thanks.

Issue Analytics

State:
Created 4 years ago
Comments:7

Top GitHub Comments

1reaction

bencrippscommented, Feb 4, 2020

We’re also seeing a memory growth issue, which only became apparent when we switched to Node 12.

However, we’re currently only running hot-shots@5.9.2. I’m not 100% sure if our memory-leak is related to hot-shots, but I’m going to collect a heap-dump, and I will report the findings when I know more.

0reactions

bdeittecommented, Mar 19, 2020

Not hearing anything new here so closing this out

Top Results From Across the Web

Node v12.16.2 memory issues after upgrade from v8 ... - GitHub

It seems like the leak is coming from getObjectFromJson. If you comment out the body on that function memory doesn't seem to grow...

[NODE-2460] Memory Leak with ReplSet and NodeJS v12

Bug - A problem which impairs or prevents the functions of the product. NODE-3340 Memory Leak with ReplSet and NodeJS v12. Unknown -...

Memory Diagnostics - Using Heap Snapshot - Node.js

You can find a memory leak by compaing too snapshots. It's important to make sure the snapshots difference does not contain unnecessary information....

Investigating Memory Leaks with valgrind - Nodejs API 文档

Investigating Memory Leaks with valgrind. A Node.js process may run out of memory due to excessive consumption of native memory. Native Memory is...

How to Find, Fix, and Prevent Node.js Memory Leaks

Memory management for any application is essential. This post looks at what memory leaks are and how to avoid them in Node.js applications....