Memory Leak in LogWriter on AppService plan

• Timestamp: 16/4/2018
• Function App version: 1.0
• Function App name: smartanalyticsnrt-anomaly-int
• Function name(s) (as appropriate): IsAnomaly
• Invocation ID:
• Region:

Hi All,

We have been running into sporadic memory issues the past several weeks with our Azure Function- seemingly out of the blue our process starts to increasing in memory until it hits its limit and crashes.

Looking at one of the dumps, it seems that the finalizer queue is blocked on finalizing an instance of Nodejsfunc (see below), but as far as we know, this isn’t a class that we or our direct dependencies use.

Is it possible this is coming from the AF infra? Is anyone familiar with an issue like this?

Any help would be greatly appreciated.

0:012> !clrstack
OS Thread Id: 0x1af8 (12)
Child SP       IP Call Site
1e37f86c 7767e59c [InlinedCallFrame: 1e37f86c] 
1e37f864 1ede8117 *** ERROR: Symbol file could not be found.  Defaulted to export symbols for easyauthshim.dll - 
DomainBoundILStubClass.IL_STUB_PInvoke(Void**)
1e37f86c 1ede8090 [InlinedCallFrame: 1e37f86c] .uv_sem_wait(Void**)
1e37f89c 1ede8090 *** WARNING: Unable to verify checksum for edge_nativeclr.node
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for edge_nativeclr.node - 
.V8SynchronizationContext.RegisterAction(Void (Void*), Void*)
1e37f8b0 1edeaa7a NodejsFunc.!NodejsFunc()
1e37f8dc 1edea991 NodejsFunc.Dispose(Boolean)
1e37f8f8 1edea95d NodejsFunc.Finalize()

0:012> kv
# ChildEBP RetAddr  Args to Child              
00 1e37f7c8 774faf59 00001360 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])
01 1e37f83c 774faeb2 00001360 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x99 (FPO: [SEH])
*** WARNING: Unable to verify checksum for node.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for node.dll - 
02 1e37f850 5dc8768e 00001360 ffffffff 1ede8117 KERNELBASE!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
03 1e37f894 1ede8090 2769e8e8 120962b8 120962ac node!uv_sem_wait+0xe
04 1e37f8a8 1edeaa7a 2769e8e8 7348ef99 1e37f8c0 0x1ede8090
05 1e37f8d4 1edea991 00000000 00000000 00000000 0x1edeaa7a
06 1e37f8f0 1edea95d 1e37f94c 735363f2 1e37f93c 0x1edea991
07 1e37f8f8 735363f2 1e37f93c 73627020 1e37fae0 0x1edea95d
08 1e37f94c 735364e5 00000000 1ee1bd8c 11ff4a34 clr!MethodTable::FastBox+0xb0 (FPO: [Non-Fpo])
09 1e37f970 7353626c 11ff4a34 1ef1d258 11ff4a34 clr!MethodTable::CallFinalizer+0x139 (FPO: [Non-Fpo])
0a 1e37f980 735362e3 177d5762 00000000 11ff4a34 clr!CallFinalizer+0xa6 (FPO: [0,1,0])
0b 1e37f9d0 73536345 00000000 1e37f9ef 1e37fae0 clr!FinalizerThread::DoOneFinalization+0x84 (FPO: [Non-Fpo])
0c 1e37fa00 7358dbd4 7358dbc0 1e37fa20 7362fdca clr!FinalizerThread::FinalizeAllObjects+0xa6 (FPO: [0,5,4])
0d 1e37fa0c 7362fdca 1e37fc50 0554c640 1e37fdec clr!FinalizerThread::FinalizeAllObjects_Wrapper+0x14 (FPO: [Non-Fpo])
0e 1e37fa20 7362fe34 177d5476 1e37fdec 00000000 clr!Thread::DoExtraWorkForFinalizer+0x1b4 (FPO: [0,1,4])
0f 1e37fac4 7362ff01 177d5592 0554c640 1e37fdec clr!Thread::DoExtraWorkForFinalizer+0x237 (FPO: [Non-Fpo])
10 1e37fb20 7349b505 1ef1d258 1e37fbf4 7361b2c4 clr!Thread::DoExtraWorkForFinalizer+0x5fb (FPO: [Non-Fpo])
11 1e37fb2c 7361b2c4 1e37fdec 177d5546 11f1b5d4 clr!Thread::DoExtraWorkForFinalizer+0x13a (FPO: [Non-Fpo])
12 1e37fbf4 7358cb8f 1ef1d258 7349b4f0 1e37fdec clr!Thread::DoADCallBack+0x30f (FPO: [Non-Fpo])
13 1e37fc18 7358db8c 177d52da 00000000 11f1b5d4 clr!Thread::DoExtraWorkForFinalizer+0x1a2 (FPO: [0,1,4])
14 1e37fc68 73536345 00000000 1e37fc87 1e37fda0 clr!FinalizerThread::DoOneFinalization+0x129 (FPO: [Non-Fpo])
15 1e37fc9c 73535741 177d521a 1e37fda0 73535670 clr!FinalizerThread::FinalizeAllObjects+0xa6 (FPO: [0,5,4])
16 1e37fccc 7362fdca 1e37fdec 0554c640 1e37fdec clr!FinalizerThread::FinalizerThreadWorker+0xed (FPO: [1,5,0])
17 1e37fce0 7362fe34 177d5336 1e37fdec 00000000 clr!Thread::DoExtraWorkForFinalizer+0x1b4 (FPO: [0,1,4])
18 1e37fd84 7362ff01 177d5352 7354cee0 73535670 clr!Thread::DoExtraWorkForFinalizer+0x237 (FPO: [Non-Fpo])
19 1e37fde0 7354ced8 00000001 00000000 1ef1d258 clr!Thread::DoExtraWorkForFinalizer+0x5fb (FPO: [Non-Fpo])
1a 1e37fe08 7354cf9f 177d50f6 7354cee0 00000000 clr!ManagedThreadBase::FinalizerBase+0x33 (FPO: [Non-Fpo])
1b 1e37fe44 734db601 00000000 00000000 00000000 clr!FinalizerThread::FinalizerThreadStart+0xd4 (FPO: [Non-Fpo])

Thanks!

[AssafNeufeld]

UPDATE + WORKAROUND:

OK, so the above description is certainly the cause of the memory leak. We removed the app settings

“AzureWebJobsDashboard”

From our functions, thus disabling logging to Azure Storage (we only rely on AppInsights telemetry in any case). The memory leak disappeared and has not returned since (after repro-ing dozens of times over the past few months).

Thanks!

[paulbatum]

This was fixed in webjobs 2.x and 3.x, which covers all versions of functions. It was not backported to webjobs 1.x.