PythonEngine.Shutdown() / Runtime.Shutdown() slowness

Environment

• Pythonnet version: 3.0.1
• Python version: 3.9.12
• Operating System: Windows or Linux
• .NET Runtime: net6.0

Details

• Describe what you were trying to get done.

We are embedding Python in a .NET app using pythonnet.

The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.

Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().

Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.

• Minimal, Complete, and Verifiable example

using Python.Runtime;

namespace Test;

static class Program
{
    static void Main()
    {
        var mem = new List<Array>();
        Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
        PythonEngine.Initialize();
        var threadState = PythonEngine.BeginAllowThreads();

        using (Py.GIL())
        {
            using (var scope = Py.CreateScope())
            {
                scope.Set("a", 1);
                scope.Set("b", 2);
                scope.Exec("result = a + b");
                var result = scope.Get("result").As<int>();
                Console.WriteLine($"Sum = {result}");
            }
        }

        // allocate dotnet-managed memory unrelated to pythonnet
        var rnd = new Random();
        for (var i = 0; i < 100_000_000; ++i)
        {
            mem.Add(new byte[rnd.Next(1, 10)]);
        }

        PythonEngine.EndAllowThreads(threadState);
        Console.WriteLine("Shutdown start");
        var start = DateTime.Now;
        PythonEngine.Shutdown();
        var diff = DateTime.Now - start;
        Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
    }
}

Typical output:

Sum = 3
Shutdown start
Shutdown stop 61.6796983

[filmor]

@AlexGilmor If you want to shut down Python, use Shutdown. The slowness is due to to multiple garbage collection rounds and (potentially contributing, but not the only factor) the runtime stashing that was introduced to allow for AppDomain switching.

@nlogozzo Since you are not actually using the stashing, you can try setting a no-op formatter:

public class NoopFormatter : IFormatter {
  public object Deserialize(Stream s) => throw new NotImplementedException();
  public void Serialize(Stream s, object o) {}

  public SerializationBinder Binder { get; set; }
  public StreamingContext Context { get; set; }
  public ISurrogateSelector SurrogateSelector { get; set; }
}

Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);

[kikaragyozov]

Environment

• Pythonnet version: 3.0.1
• Python version: 3.9.12
• Operating System: Windows or Linux
• .NET Runtime: net6.0

Details

• Describe what you were trying to get done.

We are embedding Python in a .NET app using pythonnet.

The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.

Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().

Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.

• Minimal, Complete, and Verifiable example

using Python.Runtime;

namespace Test;

static class Program
{
    static void Main()
    {
        var mem = new List<Array>();
        Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
        PythonEngine.Initialize();
        var threadState = PythonEngine.BeginAllowThreads();

        using (Py.GIL())
        {
            using (var scope = Py.CreateScope())
            {
                scope.Set("a", 1);
                scope.Set("b", 2);
                scope.Exec("result = a + b");
                var result = scope.Get("result").As<int>();
                Console.WriteLine($"Sum = {result}");
            }
        }

        // allocate dotnet-managed memory unrelated to pythonnet
        var rnd = new Random();
        for (var i = 0; i < 100_000_000; ++i)
        {
            mem.Add(new byte[rnd.Next(1, 10)]);
        }

        PythonEngine.EndAllowThreads(threadState);
        Console.WriteLine("Shutdown start");
        var start = DateTime.Now;
        PythonEngine.Shutdown();
        var diff = DateTime.Now - start;
        Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
    }
}

Typical output:

Sum = 3
Shutdown start
Shutdown stop 61.6796983

Your issue does not occur when the project is ran in .NET 7 using a Release configuration. When ran in Debug configuration, the shutdown does indeed slow down significantly - from 0.08 seconds up to 63 seconds.

public class NoopFormatter : IFormatter { public object Deserialize(Stream s) => throw new NotImplementedException(); public void Serialize(Stream s, object o) {}

public SerializationBinder Binder { get; set; } public StreamingContext Context { get; set; } public ISurrogateSelector SurrogateSelector { get; set; } }

Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);

This does not fix the issue when the code is ran in Debug configuration under .NET 7.