Memory leak on ARM (Raspbian) with .NET Core 3.1

Hi,

I have a memory leak since migration my application to .NET Core 3.1.

Environment

• Raspbian (9)
• Raspberry Pi 3 B+
• .NET Core 3.1 Self contained application in Release Build (all latest SDK and nugets)
• IronPython 2.7.10-candiate1 (but problems also present in 2.7.9).

Problem The software is a smart home system (https://github.com/chkr1011/Wirehome.Core) and runs 24/7 on the Raspberry Pi. With .NET Core 2.1 and 2.2 I had no issues with memory usage. It stays at 300 MB for several weeks or even months. But since migrating to .NET Core 3.1 and latest IronPython the system consumes all memory after roughly 3 days so that it stops working and I have to reboot the system.

Sample The following code shows basically what I am doing. It calls some Python methods and Python methods can register callbacks which are invoked from my host from different threads (Tasks). This sample code runs without problems on my PC but it seems that the ARM version has a leak somehow.

I the live project the Python scripts are also called within ASP.MVC requests, MQTT messages etc. So many different threads. I also have around 600 active ScriptScopes in memory which are invoked via previously registered callbacks or have predefined methods which are called from some APIs etc. Also data is send via some PythonDictionary across these ScriptScopes.

List<Action<PythonDictionary>> _callbacks = new List<Action<PythonDictionary>>();

void Main()
{
	var engine = IronPython.Hosting.Python.CreateEngine();
	var scriptScope = engine.CreateScope();
	var source = scriptScope.Engine.CreateScriptSourceFromString(GetPythonCode(), SourceCodeKind.File);
	var compiledCode = source.Compile();
	compiledCode.Execute(scriptScope);
	
	IDictionary<string, object> wirehomeWrapper = new ExpandoObject();
	wirehomeWrapper["ping"] = new Action<PythonDictionary>(Ping);
	wirehomeWrapper["register_callback"] = new Action<Action<PythonDictionary>>(RegisterCallback);
	
	scriptScope.SetVariable("wirehome", wirehomeWrapper);
	
	var testPythonFunction = (PythonFunction)compiledCode.Engine.Operations.GetMember(scriptScope, "test");
	var initPythonFunction = (PythonFunction)compiledCode.Engine.Operations.GetMember(scriptScope, "init");

	compiledCode.Engine.Operations.Invoke(initPythonFunction, new PythonDictionary
	{
		["x"] = 1,
		["y"] = "Test string"
	});

	while(true)
	{
		Task.Run(() =>
		{
			compiledCode.Engine.Operations.Invoke(testPythonFunction, new PythonDictionary
			{
				["x"] = 1,
				["y"] = "Test string"
			});

			foreach (var callback in _callbacks)
			{
				callback(new PythonDictionary
				{
					["x"] = 1,
					["y"] = "Test string"
				});
			}
		}).Wait();
		
		Thread.Sleep(10);
	}
}

void Ping(PythonDictionary parameters)
{
	Console.Write(".");
}

void RegisterCallback(Action<PythonDictionary> callback)
{
	_callbacks.Add(callback);
}

string GetPythonCode()
{
	return @"
call_count = 0

def init(params):
	wirehome.register_callback(test2)

def test(params):
	global call_count
	call_count = call_count + 1
	
	#wirehome.ping({'x': 1})
	
def test2(params):
	test(params)
";
}

Memory dump I tried to create a memory dump using dotnet-dump but I did not find anything useful. Maybe the following screenshot contains something useful.

I also tried to use LightweightScopes or GC.Collect() but with no difference. What bothers me is that it was running with .NET Core 2.2. Nothing special has changed in the migration to .NET Core 3.1 (at least from my perspective).

This issue is probably related to #688

Please let me know if I can provide some more details or test some workarounds etc. At the moment I have to restart my entire house every 3 days 😅.

Best regards Christian

[chkr1011]

It seems that the issue is coming from somewhere else in my code. I am not 100 % percent sure but after deactivating some parts which are starting processes through C# (Process.Start) the memory leak is gone. I checked the code a hundred times but saw now potential issue like missing Dispose etc.

At the moment I am rewriting the code to use the Popen method from the Python standard library. I will see what happens. The code is actually running the app “coap-client” every two seconds. That lead to the memory leak. I will let you know as soon as I got some results.

[chkr1011]

OK now I got it. I will test a build with the binaries from the directory netcoreapp2.1 from latest 2.7.10-candidate1 ZIP package. If you prefer another version please let me know. I may need a couple of days to see if something has changed…