Terminating connection due to administrator command

This occurred during a service restart of our postgres master. A .NET process reboot was the only method to solve it (a connection pool reset isn’t something we expose as an api).

Npgsql.PostgresException (0x80004005): 57P01: terminating connection due to administrator command
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming)
   at Npgsql.NpgsqlCommand.ExecuteReaderAsync(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
   at Dapper.SqlMapper.MultiMapAsync[TFirst,TSecond,TThird,TFourth,TFifth,TSixth,TSeventh,TReturn](IDbConnection cnn, CommandDefinition command, Delegate map, String splitOn) in C:\projects\dapper\Dapper\SqlMapper.Async.cs:line 956
--- End of stack trace from previous location where exception was thrown ---
   at Crowded.Mediator.Mediator.Send[TRequest,TResponse](TRequest request, CancellationToken cancellationToken) in /_/Crowded.Mediator/Mediator.cs:line 182
   at lambda_method(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ExceptionContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Ply.TplPrimitives.UnitTaskAwaiterMethods`1.Ply-TplPrimitives-IAwaiterMethods`2-GetResult(TaskAwaiter& awt)
   at Ply.TplPrimitives.TplAwaitable`4.GetNext()
   at [e-mail]()
--- End of stack trace from previous location where exception was thrown ---
   at [e-mail]()
   at Ply.TplPrimitives.ContinuationStateMachine`1.System-Runtime-CompilerServices-IAsyncStateMachine-MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Ply.TplPrimitives.UnitTaskAwaiterMethods`1.Ply-TplPrimitives-IAwaiterMethods`2-GetResult(TaskAwaiter& awt)
   at Ply.TplPrimitives.TplAwaitable`4.GetNext()
   at [e-mail]()
  Exception data:
    Severity: FATAL
    SqlState: 57P01
    MessageText: terminating connection due to administrator command
    File: postgres.c
    Line: 2913
    Routine: ProcessInterrupts

Related: https://github.com/npgsql/npgsql/issues/2809

We must be missing a mechanism to clean up connections that break in this specific way.

[Plasma]

Hi @roji I’m another user of pgsql and I can reproduce the application being in a (temporary) state where after some maintenance event (eg, all pooled connections had been closed due to server-side disconnect), the application keeps drawing (closed/dead) connections from the pool, until the pool is empty, and new connections work.

In my quick testing, it seems there can be ~20 connections after immediate startup that it will need to cycle through before it will try creating a new connection (that succeeds).

Repro steps

1. Have your app perform a query
2. Kill all connections via a management command:

select pg_terminate_backend(pid) from pg_stat_activity where datname= current_database()
and usename = 'app_username'
and pid <> pg_backend_pid()

1. Have your app try querying again, it should now get a PostgresException of 57P01: terminating connection due to administrator command

2. Keep trying and wait for the pool to be cleared from each connection attempt, after 20-40 tries it seems to work again.

Workaround

Catch this particular error and pre-emptively clear the pool:

// In your global exception handler
// If this is a PostgresException and appears to be a connection issue, clear connection pools in case connections have been severed
if (exception is PostgresException {SqlState: "57P01"} /* 57P01: terminating connection due to administrator command */)
	NpgsqlConnection.ClearAllPools();

The next query will work right away (instead of waiting 20-40 queries).

Thoughts

According to SqlClient (SQL Server) at https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-connection-pooling?redirectedfrom=MSDN the driver will not use connections it knows are severed, if possible (eg, perhaps the socket has been closed)?

I wonder if Npgsql could detect closed sockets (if this is possible, and the potential issue here) and not return them from the connection pool for use?

[glebov21]

I have this issue too. @codewithtomi can you try add “Pooling=false;” into connection string? Error gone? If true, i think what pg_terminate_backend make pool broken.

Solution: Use ClearPool instead of pg_terminate_backend:

Npgsql.NpgsqlConnection.ClearPool(new NpgsqlConnection(connectionDb) as Npgsql.NpgsqlConnection);