ConfigurationRoot holding too many ConfigurationReloadToken - Possible memory leak
See original GitHub issueWe have built Microservice using NETCORE 2.2. On load test we noticed a significant memory growth. The Dump shows below:
Memory dump Analysis: Seems “System.Threading.CancellationTokenSource+CallbackNode” was being repeated called for large amount of times. While each of them need to hold a System.Threading.IAsyncLocal[] object. All these calls can be traced back to Microsoft.Extensions.Configuration.ConfigurationReloadToken.
0:000> !gcroot 53184fec
Thread 31f0:
0057E9F4 6A3511DA System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) [E:\A\_work\367\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2959]
ebp+28: 0057e9fc
-> 03313554 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Microsoft.AspNetCore.Hosting.WebHostExtensions+<RunAsync>d__4, Microsoft.AspNetCore.Hosting]]
. . .
. . .
. . .
-> 0322D2EC System.Func`2[[Microsoft.ApplicationInsights.Extensibility.ITelemetryProcessor, Microsoft.ApplicationInsights],[Microsoft.ApplicationInsights.Extensibility.ITelemetryProcessor, Microsoft.ApplicationInsights]][]
-> 0322C920 System.Func`2[[Microsoft.ApplicationInsights.Extensibility.ITelemetryProcessor, Microsoft.ApplicationInsights],[Microsoft.ApplicationInsights.Extensibility.ITelemetryProcessor, Microsoft.ApplicationInsights]]
-> 0322861C Microsoft.DiagnosticServices.SnapshotCollector.HostingStartup.Startup+SnapshotCollectorTelemetryProcessorFactory
-> 031FE110 Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope
-> 032055E8 System.Collections.Generic.List`1[[System.IDisposable, System.Private.CoreLib]]
-> 03540334 System.IDisposable[]
-> 03204660 Microsoft.Extensions.Logging.Console.ConsoleLoggerProvider
. . .
. . .
. . .
-> 0320407C Microsoft.Extensions.Configuration.ConfigurationRoot
-> 03204018 System.Collections.Generic.List`1[[Microsoft.Extensions.Configuration.IConfigurationProvider, Microsoft.Extensions.Configuration.Abstractions]]
-> 0320403C Microsoft.Extensions.Configuration.IConfigurationProvider[]
-> 03204030 Microsoft.Extensions.Configuration.ChainedConfigurationProvider
-> 03203ED0 Microsoft.Extensions.Configuration.ConfigurationSection
-> 0312F89C Microsoft.Extensions.Configuration.ConfigurationRoot
-> 0312F8AC Microsoft.Extensions.Configuration.ConfigurationReloadToken
-> 0312F8B8 System.Threading.CancellationTokenSource
-> 031EB964 System.Threading.CancellationTokenSource+CallbackPartition[]
-> 031EB974 System.Threading.CancellationTokenSource+CallbackPartition
-> 782E56C0 System.Threading.CancellationTokenSource+CallbackNode
. . .
. . .
. . .
-> 531875D0 System.Threading.CancellationTokenSource+CallbackNode
-> 531874DC System.Threading.ExecutionContext
-> 53184FEC System.Threading.IAsyncLocal[]
I found the similar bug been reported here: https://github.com/aspnet/Extensions/issues/861
However, this bug seems to only been fixed in NETCORE 3.0.0.
Given that we are facing this issue on NETCORE 2.2, can you please advice:
- Whether this fix has been back ported to NETCORE 2.2 or less?
- IF not how do we apply this fix on NETCORE2.2 without having to upgrade to 3.0.0.
We also tried setting the disabling reload but didn’t help.
Issue Analytics
- State:
- Created 4 years ago
- Comments:5 (2 by maintainers)
Top Results From Across the Web
asp.net core - NETCORE ConfigurationRoot Memory Leak
We are facing memory leak issue with ConfigurationProviders ConfigurationRoot. enter image description here. Looks like this issue may have been ...
Read more >8 Ways You can Cause Memory Leaks in .NET
There are 2 related core causes for this. The first core cause is when you have objects that are still referenced but are...
Read more >Memory management and patterns in ASP.NET Core
NET. Analyzing and understanding memory issues can be challenging. This article: Was motivated by many memory leak and GC not working issues.
Read more >Possible memory leak. : r/Archero
The problem: When I am playing sometimes the game gets laggy and laggy like being dragged and, when that lag gets too hard,...
Read more >Memory leak : r/csharp
The issue is that RAM usage is constantly increasing too, so the application crashes every certain time. I use the 'using' statement with ......
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
As part of the migration of components from dotnet/extensions to dotnet/runtime (https://github.com/aspnet/Announcements/issues/411) we will be bulk closing some of the older issues. If you are still interested in having this issue addressed, just comment and the issue will be automatically reactivated (even if you aren’t the author). When you do that, I’ll page the team to come take a look. If you’ve moved on or workaround the issue and no longer need this change, just ignore this and the issue will be closed in 7 days.
If you know that the issue affects a package that has moved to a different repo, please consider re-opening the issue in that repo. If you’re unsure, that’s OK, someone from the team can help!
Thanks heaps for your help @davidfowl on helping us identify the root cause of memory leak.
For others benefit, see screenshot below on cause of steady memory growth. Serilog was configured as Transient as opposed to Singleton. Hence it was raising too many Reload Token calls.