CancellationToken registration leak, regression

Describe the bug

After updating to SqlClient 5.0.1, we observe a memory leak in our application. Memory dumps show a huge amount of CancellationTokenSource+CallbackNode entries (we had a dump with 2 GB of those), referencing SqlCommand objects.

We believe this might be due to the changes in:

• https://github.com/dotnet/SqlClient/pull/1785 for 5.0.1 backport
• https://github.com/dotnet/SqlClient/pull/1781 for 5.1.0

With the changes in these PRs, it seems the CancellationToken registration is not always disposed. There are several if blocks that may return before the registration object gets to be saved in the context. And exceptions could also be thrown.

To reproduce

using Microsoft.Data.SqlClient; // 5.0.1
using System.Threading;
using System.Threading.Tasks;

public static class Program
{
    public static async Task Main()
    {
        const string query = "SELECT 1";

        // this could be the application lifetime cancellation
        // or another form of global shared cancellation
        using var longRunningCts = new CancellationTokenSource();

        var connectionString = new SqlConnectionStringBuilder
        {
            DataSource = "(local)",
            IntegratedSecurity = true,
            TrustServerCertificate = true,
        }.ToString();

        try
        {
            using (var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(longRunningCts.Token))
            using (var connection = new SqlConnection(connectionString))
            {
                await connection.OpenAsync(linkedCts.Token);

                using (var command = new SqlCommand(query, connection))
                using (var reader = await command.ExecuteReaderAsync(linkedCts.Token))
                {
                    linkedCts.Cancel();

                    while (await reader.ReadAsync(linkedCts.Token))
                    {

                    }
                }
            }
        }
        catch (Exception e)
        {

        }

        // => attach a debugger here
        // linkedCts as well as all the SQL objects have been disposed above
        // however a memory snapshot shows that the CancellationTokenSource+Registrations and CancellationTokenSource+CallbackNode are still there, including the objects they reference
        // so garbage collection cannot happen correctly
    }
}

Expected behavior

No memory leak, CancellationTokenSource+CallbackNode entries are not present in a dump.

Further technical details

Microsoft.Data.SqlClient version: 5.0.1 .NET target: .NET 6.0.3 SQL Server version: SQL Server 2019 Operating system: Windows 2019

Additional context

[aarondandy]

I think many out there (like me) who took way too long to discover this as the root cause are going to really appreciate upgrading to 5.1.0 when it is released.

[Shane32]

I’m having this issue also. After moving to 5.0.1 my Linux server would leak memory rapidly until crashing. Analyzing a memory dump revealed millions of instances of SqlCommand rooted only by the cancellation token registration (CallbackNode). Even after just a couple hours, I can confirm that reverting to 5.0.0 has solved the problem. For an in-depth analysis, including dump screen shots, system configuration, and other information, please see this issue:

• https://github.com/linq2db/linq2db/issues/3817

(Azure App Service - Linux - Memory Working Set - last 30 days)

(Azure App Service - Linux - Memory Working Set - last 3 days)

I recommend marking 5.0.1 as unstable on NuGet for three reasons:

1. This bug potentially affects anyone that has SQL code continuously or periodically executing within a BackgroundService, which is probably common enough.
2. The bug is not necessarily obvious (if the memory leak is small), and is very difficult to track down and diagnose, and even harder to write a test for. (I spent probably 40 hours on this.)
3. The severity of the bug is such that it can often result in the application crashing. Not only that, it can crash other applications within the same server/VM. (For me, it affected all web apps within within same Azure App Service Plan.)