connection.TypeMapper.RemoveMapping / AddMapping bug

Steps to reproduce

public static class UnitTests
{
    private static string _connectionString = 
        "Server=localhost;Port=5432;Database=myDatabase;User Id=myUserName;Password=myPassword";

    public static void FailsWithConnectionSpecificMappingAndDapper()
    {
        using (NpgsqlConnection connection = new NpgsqlConnection(_connectionString))
        {
            connection.Open();

            connection.TypeMapper.RemoveMapping("text");
            connection.TypeMapper.AddMapping(new NpgsqlTypeMappingBuilder
            {
                PgTypeName = "citext",
                NpgsqlDbType = NpgsqlDbType.Citext,
                DbTypes = new[] { DbType.String },
                ClrTypes = new[] { typeof(string) },
                TypeHandlerFactory = new TextHandlerFactory()
            }.Build());

            if (!connection.Query<bool>(
                "SELECT @p = 'hello'::citext", new { p = "HeLLo" }).First()) // fails here
            {
                throw new Exception("failed"); // never makes it here
            }
        }
    }

    public static void WorksWithGlobalMappingAndDapper()
    {
        NpgsqlConnection.GlobalTypeMapper.RemoveMapping("text");
        NpgsqlConnection.GlobalTypeMapper.AddMapping(new NpgsqlTypeMappingBuilder
        {
            PgTypeName = "citext",
            NpgsqlDbType = NpgsqlDbType.Citext,
            DbTypes = new[] { DbType.String },
            ClrTypes = new[] { typeof(string) },
            TypeHandlerFactory = new TextHandlerFactory()
        }.Build());

        using (NpgsqlConnection connection = new NpgsqlConnection(_connectionString))
        {
            connection.Open();

            if (!connection.Query<bool>(
                "SELECT @p = 'hello'::citext", new { p = "HeLLo" }).First())
            {
                throw new Exception("failed");
            }
        }
    }

    public static void WorksWithConnectionSpecificMappingAndAdoNet()
    {
        using (NpgsqlConnection connection = new NpgsqlConnection(_connectionString))
        {
            connection.Open();

            connection.TypeMapper.RemoveMapping("text");
            connection.TypeMapper.AddMapping(new NpgsqlTypeMappingBuilder
            {
                PgTypeName = "citext",
                NpgsqlDbType = NpgsqlDbType.Citext,
                DbTypes = new[] { DbType.String },
                ClrTypes = new[] { typeof(string) },
                TypeHandlerFactory = new TextHandlerFactory()
            }.Build());

            using (NpgsqlCommand command = new NpgsqlCommand(
                "SELECT @p = 'hello'::citext", connection))
            {
                command.Parameters.AddWithValue("p", "HeLLo");

                if (!(bool)command.ExecuteScalar())
                {
                    throw new Exception("failed");
                }
            }
        }
    }
}

Note: Make sure you run FailsWithConnectionSpecificMappingAndDapper 1st because if WorksWithGlobalMappingAndDapper runs 1st it will mask the bug.

The issue

I’m using the citext mapping that you suggested in response to #1765 (related to #1475). I think there is a bug, but it’s a rare use case so it probably hasn’t been noticed.

Your unit test here shows how to map it at the connection-specific level. This works fine with ADO.Net, but fails with Dapper. I’ve debugged it a bit & believe it’s an Npgsql issue, not a Dapper issue. However, NpgsqlConnection.GlobalTypeMapper works with both Ado.Net & Dapper.

From stepping through the code it seems to be related to GlobalTypeMapper.Instance.ToNpgsqlDbType(value) not getting setup correctly when the mapping is changed on a local connection. I haven’t debugged any more than that.

On a side note, PostgreSQL can implicitly type-cast between text & citext if needed, so it may not be obvious that you need to remap citext in npgsql because your queries might return the correct results even without the mapping. The reason I even attempted to map citext was because my queries were extremely slow in npgsql but fast in pgAdmin. After lengthy debugging, I realized it was because my table columns were citext, but I wasn’t mapping to citext in npgsql. Maybe this could be mentioned at https://www.npgsql.org/doc/performance.html to help other people avoid my mistake?

Exception Message:

{Npgsql.NpgsqlException (0x80004005): The NpgsqlDbType 'Text' isn't present in your database. You may need to install an extension or upgrade to a newer version.

Stack Trace:

   at Npgsql.TypeMapping.ConnectorTypeMapper.GetByNpgsqlDbType(NpgsqlDbType npgsqlDbType) in C:\projects\npgsql\src\Npgsql\TypeMapping\ConnectorTypeMapper.cs:line 106
   at Npgsql.NpgsqlParameter.ResolveHandler(ConnectorTypeMapper typeMapper) in C:\projects\npgsql\src\Npgsql\NpgsqlParameter.cs:line 523
   at Npgsql.NpgsqlCommand.ValidateParameters() in C:\projects\npgsql\src\Npgsql\NpgsqlCommand.cs:line 796
   at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) in C:\projects\npgsql\src\Npgsql\NpgsqlCommand.cs:line 1141
   at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior) in C:\projects\npgsql\src\Npgsql\NpgsqlCommand.cs:line 1130
   at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)
   at Dapper.SqlMapper.ExecuteReaderWithFlagsFallback(IDbCommand cmd, Boolean wasClosed, CommandBehavior behavior) in C:\projects\dapper\Dapper\SqlMapper.cs:line 1053
   at Dapper.SqlMapper.QueryImpl[T](IDbConnection cnn, CommandDefinition command, Type effectiveType)+MoveNext() in C:\projects\dapper\Dapper\SqlMapper.cs:line 1081
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType) in C:\projects\dapper\Dapper\SqlMapper.cs:line 723

Further technical details

Npgsql version: 4.0.3 Dapper: 1.50.5 PostgreSQL version: PostgreSQL 10.1, compiled by Visual C++ build 1800, 64-bit Operating system: Windows 10 Pro 64-bot

[speige]

Elaborating on the performance issue. It’s not technically an npgsql issue, however it’s a “gotcha” a programmer might make without realizing it.

In pgAdmin, if you execute this:

--prep work
create table my_users_table ( userid citext )
create index userid_index on my_users_table (userid)
insert into my_users_table (userid) select md5(random()::text) from generate_Series(1,1000000) -- will take a few min
select * from my_users_table order by random() limit 1 -- use this value for following queries

--monitor the ms when running these queries as well as the execution plans
SELECT * from my_users_table where userid = 'VALUE_FROM_ABOVE'::citext -- index only scan (fast)
SELECT * from my_users_table where userid = 'VALUE_FROM_ABOVE'::text -- same as above
SELECT * from my_users_table where userid = 'VALUE_FROM_ABOVE' -- seq scan (slow)

You’ll notice that if the column & parameter don’t match, postgres is choosing to type-cast every value in the table to a ::text rather than type-casting the parameter to a ::citext. This requires a full table scan which is much slower than an index lookup. If the data & where clause happen to be the same-case, the programmer will still get the results they expect & not realize there is significant slowness due to poor execution plan.

For this reason, when doing a WHERE clause on a citext column, it’s important to modify the mappings in npgsql to ensure you’re sending a citext parameter, not a text parameter. If you’re creating NpgsqlParameters directly in code this should be obvious. However, most people use shorthand code that auto-generates the NpgsqlParameter behind the scenes (Parameters.AddWithValue or Dapper), so they don’t think about whether the underlying parameter was creating as text or citext.

[roji]

First, mapping string to citext as shown above is probably a bad idea - unless your entire application/database is always case-insensitive, you will have trouble dealing with non-insensitive strings, as this affects all strings written/read. So this should be discouraged except for some pretty specific rare cases…

You may also be interested in https://github.com/StackExchange/Dapper/issues/433 and specifically https://github.com/StackExchange/Dapper/pull/471 for setting string->citext mapping at the Dapper level (instead of at the ADO.NET level).

Regardless, the analysis seems to be correct: when converting between DbType and NpgsqlDbType on NpgsqlParameter, the global type mapper is used. As @austindrenski wrote, this is because the ADO.NET API (unfortunately) allows you to have detached DbParameters, which haven’t been added to any to any specific command, and therefore which aren’t associated to any specific connection. We are still expected to do various translations on these parameters (e.g. infer an NpgsqlDbType from a set Value), so the only option we have is to use the global type mapper. However, as you pointed out when a command is actually executed, parameters are bound using the command’s connection’s type mapper.

Looking at this again, I’m unsure on why we eagerly do NpgsqlDbType/DbType conversions when one of them is set, rather than lazily. In other words, when DbType is set, NpgsqlDbType could simply be set to null, and NpgsqlDbType’s getter would do the DbType->NpgsqlDbType lookup if required. This would solve this issue, and also be slightly better for perf as useless translations are avoided.

When doing the translation, we could optionally also check whether the parameter is associated to a command which itself is associated with a connection, and if so, use that connection’s type mapper instead of the global type mapper. But assuming we make the translations lazy as described above, this doesn’t really seem to be needed.

Makes sense?