Consider reclaiming connections that get abandoned to the garbage collector without being disposed

The issue

The SQL server .NET client (both Microsoft.Data.SqlClient and System.Data.SqlClient) has a feature where connections that get abandoned can be GC’d and ultimately reclaimed by the connection pool even if they are not Disposed().

While obviously developers should be careful to dispose all connections, this behavior is a nice fallback because it helps a large and complex app be better able to recover from cases where connection lifetimes are mis-managed. I thought that maybe the connection pruning feature would provide this protection, but it doesn’t seem to cover quite the same cases or perhaps I just don’t know how to make it work.

Steps to reproduce

Here is an NUnit test which shows how Npgql and SqlClient handle the same situation differently.

        private void TestPoolExhaustionWithConnectionAbandonment(
            Func<int, DbConnection> dbConnectionFactory, 
            Action clearAllPools)
        {
            const int MaxPoolSize = 10;

            List<WeakReference>? CreateConnections()
            {
                var connections = new List<DbConnection>();
                for (var i = 0; i < MaxPoolSize; ++i)
                {
                    var connection = dbConnectionFactory(MaxPoolSize);
                    try { connection.Open(); }
                    catch { break; }
                    connections.Add(connection);
                }

                var openTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(5));
                Console.WriteLine(Assert.Catch<Exception>(() => dbConnectionFactory(MaxPoolSize).OpenAsync(openTimeout.Token).Wait()));

                return connections.Select(c => new WeakReference(c)).ToList();
            }

           // use up all connections in the pool
            var weakConnections = CreateConnections();

            // attempt to reclaim via GC/clear pool
            GC.Collect();
            GC.WaitForPendingFinalizers();
            clearAllPools();
            GC.Collect();
            GC.WaitForPendingFinalizers();
            // true for SqlClient, false for Npgsql
            Console.WriteLine("connections GC'd? " + !weakConnections.Any(w => w.IsAlive)); 

            var openTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(31));
            // throws: connector pool exhausted for Npgsql. Successfully opens for SqlClient
            Assert.DoesNotThrow(() => dbConnectionFactory(MaxPoolSize).OpenAsync(openTimeout.Token).Wait());
        }

        [Test]
        public void TestNpgsqlDoesNotReclaimAbandonedConnections()
        {
            this.TestPoolExhaustionWithConnectionAbandonment(
                poolSize => new NpgsqlConnection(new NpgsqlConnectionStringBuilder()
                {
                    ...
                    MaxPoolSize = poolSize,
                    // I tried various values for this, including the defaults
                    ConnectionIdleLifetime = 10,
                    ConnectionPruningInterval = 1
                }.ConnectionString),
                NpgsqlConnection.ClearAllPools
            );
        }

        [Test]
        public void TestSqlDoesNotReclaimAbandonedConnections()
        {
            this.TestPoolExhaustionWithConnectionAbandonment(
                poolSize => new SqlConnection(new SqlConnectionStringBuilder(Sql.ConnectionStringProvider.ConnectionString)
                {
                    MaxPoolSize = poolSize,
                }.ConnectionString),
                SqlConnection.ClearAllPools
            );
        }

Further technical details

For context, the way SqlClient implements this is to pass the SqlConnection object to the pool when claiming an “internal” connection. Each internal connection holds a WeakReference to the externally-exposed SqlConnection object that is currently using it. The pool maintains a list of all the internal connections, and can detect when the outer SqlConnection has been abandoned by checking the weak reference. See https://referencesource.microsoft.com/#System.Data/fx/src/data/System/Data/ProviderBase/DbConnectionPool.cs,49b6c33621853a98,references

Npgsql version: 4.1.3.1 PostgreSQL version: 12 Operating system: Windows 10

[roji]

@baal2000 no, I didn’t get around to writing a doc page on pooling. However, the requirement to dispose IDisposable objects is general in .NET, and all the code samples show usage of connections with using. I doubt documenting the need to dispose would actually prevent to accidental leaks - but we should add more docs around this at some point.

[roji]

Connection pruning is a different feature - it’s about closing idle physical connections which are in the pool (and therefore which have been properly disposed by the user).

I don’t think this is something we should implement in Npgsql. As you wrote above, it really is the user’s responsibility to dispose connections; there are many other types of disposable resources in .NET which do not automatically track and reclaim undisposed instances - I’m not sure why database connections would be an exception. Also, SqlClient’s reclaiming technique isn’t free in terms of perf - going over the list of internal connections to check the weak reference (and even just maintaining the weak references) are operations which slow down general usage for everyone, even if they’re correctly disposing their connections. That doesn’t seem right. It also seems better for the connections to leak - hopefully bringing the bug to the user’s attention quickly - rather than covering over it this way.

At the end of the day I’d rather not complicate the driver and compromise perf to workaround what is essentially a user programming bug… Let me know if this sounds convincing or you see it differently.