React to invaildated prepared statements (0A000: cached plan must not change result type)

First off, thanks for your work on this project! It’s great, and specifically auto-prepare makes our app’s queries something like 10x-100x faster. However today when deploying to prod we noticed a new type of error pop up, that can maybe be automatically retried/handled by Npgsql.

Steps to reproduce

• Have an app that uses prepared statements
• Run a migration that changes the data type of a column used by a prepared statement

The issue

I would want the library to automatically re-try a query if it gets this error message. It’s mentioned a bit here https://github.com/npgsql/npgsql/issues/1237#issuecomment-238721347, specifically

If application is doing “alter table … add column”, a server-prepared statement that does “select * from …” would fail with “not implemented” / “cached plan must not change result type”

and it also links to an example of a similar fix in pgjdbc: https://github.com/pgjdbc/pgjdbc/pull/451/files#diff-fb626514a44e1fd93551464de0ba369def2b0513a7232ce12d9c3040ea98d211R336-R359

The following exception is thrown when running the query after the migration occurs:

Exception message: 0A000: cached plan must not change result type
Stack trace:
Npgsql.PostgresException (0x80004005): 0A000: cached plan must not change result type
   at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Ardalis.Specification.EntityFrameworkCore.RepositoryBase`1.ListAsync(ISpecification`1 specification, CancellationToken cancellationToken)
   at Infrastructure.Domain.Purchases.PurchaseEventRepository.List(String purchaseId) in /src/src/Infrastructure/Domain/Purchases/PurchaseEventRepository.cs:line 68
   at ApplicationServices.PurchaseEvents.AutoClosePurchaseEvents.With(V4Purchase purchase) in /src/src/ApplicationServices/PurchaseEvents/AutoClosePurchaseEvents.cs:line 33
   at ApplicationServices.Purchases.CreatePurchase.With(V4Purchase request, AuctionHub auctionHub) in /src/src/ApplicationServices/Purchases/CreatePurchase.cs:line 64
   at TryAsyncExtensions.Try[T](TryAsync`1 self)
  Exception data:
    Severity: ERROR
    SqlState: 0A000
    MessageText: cached plan must not change result type
    File: plancache.c
    Line: 724
    Routine: RevalidateCachedQuery
   --- End of inner exception stack trace ---

Further technical details

Npgsql version: Npgsql 5.0.5, Npgsql.EntityFrameworkCore.PostgreSQL 5.0.6 PostgreSQL version: 14 Operating system: Linux (Debian)

Other details about my project setup: We have MaxAutoPrepare = 30 and AutoPrepareMinUsages = 1 in our configuration as we have a small number of queries that are run a lot and have a huge prepare cost.

Is there any appetite to implement something similar to what pgjdbc does to detect this failure case and re-parse the query? This kinda makes it really hard for us to do zero-downtime schema migrations, as apparently even just increasing the length of a varchar (which is really a no-op) will cause this exception.

[roji]

I like the idea of Polly…we could introduce a middleware that retries the whole request/unit of work for only these exceptions. Or as you also suggested, turning off prepared statements, THEN running the migration, and then finally re-enabling it.

This would give you automatic retries for other situations as a bonus, e.g. if your PG is down for a bit. I’d generally recommend a robust retry strategy (via Polly or other) to any serious production app that tries to be zero-downtime. Note that if you’re using EF Core, it comes built-in with one as well…

does the driver deallocate a prepared statement if it fails due to these errors? For example, if I wrapped the transaction in a retry middleware, would it actually succeed the 2nd time or would it keep trying to use the bad prepared statement and I have to deallocate it myself?

That’s a good point - currently it does not, and we could do something about it… We can either do an actual roundtrip for DEALLOCATE, or somehow flag the prepared statement internally as “bad” so that it doesn’t get used and gets replaced the next time a slot is needed. Let’s keep this issue for that (though unfortunately I doubt I’ll get around to it soon, with everything going on).

[roji]

The fix here would be to add an “invalidated” flag to the prepared statement, which we set when we get this error. There’s no need to DEALLOCATE right away; simply the next time that prepared statement is executed, we re-prepare it via the normal flow.

Note that for explicitly prepared commands, we’ll need to somehow reprepare as well.