IMethodCallTranslatorPlugin vs SqlNullabilityProcessor

Hi!

I have a project that is currently using EF Core 3.1 and within I have some custom RelationalTypeMapping along with implementation of IMethodCallTranslatorPlugin for some custom linq methods. Now I need to upgrade project to use EF Core 5.x and have an issue with that.

Plugin still works but when executed I get following exception:

System.InvalidOperationException: Unhandled expression '[MyCustomExpression]' of type 'MyCustomExpression' encountered in 'SqlNullabilityProcessor'.
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.VisitCustomSqlExpression(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean& nullable)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Query.Internal.NpgsqlSqlNullabilityProcessor.VisitCustomSqlExpression(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean& nullable)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean preserveNonNullableColumns, Boolean& nullable)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.VisitSqlBinary(SqlBinaryExpression sqlBinaryExpression, Boolean allowOptimizedExpansion, Boolean& nullable)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean preserveNonNullableColumns, Boolean& nullable)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.VisitSqlBinary(SqlBinaryExpression sqlBinaryExpression, Boolean allowOptimizedExpansion, Boolean& nullable)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean preserveNonNullableColumns, Boolean& nullable)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean& nullable)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SelectExpression selectExpression)
   at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Process(SelectExpression selectExpression, IReadOnlyDictionary`2 parameterValues, Boolean& canCache)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Query.Internal.NpgsqlParameterBasedSqlProcessor.ProcessSqlNullability(SelectExpression selectExpression, IReadOnlyDictionary`2 parametersValues, Boolean& canCache)
   at Microsoft.EntityFrameworkCore.Query.RelationalParameterBasedSqlProcessor.Optimize(SelectExpression selectExpression, IReadOnlyDictionary`2 parametersValues, Boolean& canCache)
   at Microsoft.EntityFrameworkCore.Query.Internal.RelationalCommandCache.GetRelationalCommand(IReadOnlyDictionary`2 parameters)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

The piece of EF’s code that’s causing this is in SqlNullabilityProcessor. First the SqlExpression Visit(SqlExpression, bool, bool, out bool) method goes through expression types, doesn’t find (obviously) my type (which inherits from SqlExpression) and defaults back to calling VisitCustomSqlExpression(SqlExpression, bool, out bool) which just throws an exception.

I get the idea that first and foremost this is for db providers so they extend the SqlNullabilityProcessor and bake in additional per-db methods, like Postgres guys does for example, cool. But it looks like it breaks “casual” method translators plugin system as I don’t see any way to register my IMethodCallTranslatorPlugin, let it return SqlExpression and play nicely with SqlNullabilityProcessor without need to introduce own overload of that class that would replace one used by default. Or am I missing something and there is a way?

[smitpatel]

Adding a custom SQL expression requires

• Creation of expression, non-provider just can new-up if they want.
• Assignment of type mapping properly to component and expression itself. This requires access to services and becomes little trickier. ISqlExpressionFactory is currently responsible for it.
• All visitors which processes Sql tree in some form which may need to interact with it. SqlNullabilityProcessor is just one of them and only one which throws but there can be side-effects.
• Finally, QuerySqlGenerator which needs to know how to print out the custom expression in actual SQL.

While the plug-in mechanism is good for non-providers to add custom translation, adding plug-in mechanism to each component above and all related visitor is a lot more maintenance code and more importantly, a huge perf penalty. For something like method translations it happens only once during compile time so fine to have. But other visitors like SqlNullability and SqlGenerator, which can occur more frequently that is not good. Also in design, SQL tree is representation of SQL supported by the database itself so if there is a construct which is valid SQL but not available in SQL tree, then should contact provider to add it.

[roji]

@EvilVir user method translators are (almost) never expected to need to produce custom expression types: users typically define translators to translate from .NET methods to existing expression types provided by EF Core (or the specific provider). Out of curiosity, can you provide some details on what new expression type you’re defining?

Note that if you just want to define new method translations, then using HasDbFunction should be easier and more lightweight than setting up your own IMethodCallTranslationPlugin - just one call in your OnModelCreating is enough. IMethodCallTranslationPlugin can be useful if you’re creating a plugin to be used in many projects.

Finally, if you do find yourself needing an expression type, then it should be quite easy to extend your provider’s SqlNullabilityProcessor, and then override VisitCustomSqlExpression to handle your new expression type (don’t forget to call base.VisitCustomSqlExpression though!).