linq2db.SQLite.MS: variety of problems caused by wrapping values in DateTime() function

Hello again.

The first problem is that unfortunately, 3.1.0 doesn’t fix the problem I had previously in #2346. When setting up my DataConnection like

            var ms = new MappingSchema();

            ms.SetDataType(typeof(DateTime), DataType.Int64);
            ms.SetConvertExpression(
                (long ticks) => new DateTime(ticks, DateTimeKind.Unspecified)
            );
            ms.SetConvertExpression(
                (DateTime dt) => dt.Ticks
            );
            ms.SetValueToSqlConverter(
                typeof(DateTime),
                (sb, dType, v) => sb.Append(((DateTime) v).Ticks.ToString(CultureInfo.InvariantCulture))
            );

            await using var conn = new DataConnection(ProviderName.SQLiteMS, connectionString, ms)
            {
                InlineParameters = true
            };

I still end up getting queries like

SELECT
        [x].[DateTime],
        [x].[Value]
FROM
        [fae5697f79034df99e0ad32292446f5a_testOut] [x]
WHERE
        DateTime([x].[DateTime]) > DateTime(637304378093620000) AND
        DateTime([x].[DateTime]) < DateTime(637304382196380000)

produced (which will always return 0 rows), whereas I should be getting

SELECT
        [x].[DateTime],
        [x].[Value]
FROM
        [fae5697f79034df99e0ad32292446f5a_testOut] [x]
WHERE
        [x].[DateTime] > 637304378093620000 AND
        [x].[DateTime] < 637304382196380000

The second problem is that even when not trying to store my DateTime values as an Int64, and just using this to set up my DataConnection

            await using var conn = new DataConnection(ProviderName.SQLiteMS, connectionString)
            {
                InlineParameters = true
            };

wrapping the values in the DateTime() function also causes indexes to not work, which is a huge performance issue for me. It produces queries like

SELECT
        [x].[DateTime],
        [x].[Value]
FROM
        [d7e9196fe84847faabf1a02518040951_testOut] [x]
WHERE
        DateTime([x].[DateTime]) > DateTime('2020-08-27 19:57:00.035') AND
        DateTime([x].[DateTime]) < DateTime('2020-09-04 01:06:43.64')

whereas if it produced queries like

SELECT
        [x].[DateTime],
        [x].[Value]
FROM
        [d7e9196fe84847faabf1a02518040951_testOut] [x]
WHERE
        [x].[DateTime] > '2020-08-27 19:57:00.035' AND
        [x].[DateTime] < '2020-09-04 01:06:43.64'

indexing would work (as seen if you run EXPLAIN QUERY PLAN on both of these). For my use-case when working with large sets of data and filtering for a small subset, the first statement takes several seconds to run and the second one takes just milliseconds. They both return the same number of rows.

Here are the query plans below:

For the first one (that Linq2Db currently generates):

SCAN TABLE d7e9196fe84847faabf1a02518040951_testOut AS x

For the second, preferred one:

SEARCH TABLE d7e9196fe84847faabf1a02518040951_testOut AS x USING INDEX idx_d7e9196fe84847faabf1a02518040951_testOut_DateTime (DateTime>? AND DateTime<?)

From looking at https://www.sqlite.org/lang_datefunc.html the function seems to exist for converting between different DateTime formats, and when I take the DateTime() function out of the queries (whether storing the DateTime as a string as it is by default, or as an Int64 as I’m doing above) it seems to work perfectly fine and with indexing working. This suggests to me that the solution is to just stop wrapping DateTime values in the DateTime() function, unless I’m missing some important reason for doing so?

linq2db v3.1.0 linq2db.SQLite.MS Ubuntu 20.04 .NET Core 3.1.7

[MaceWindu]

Probably we can only recommend you to use custom build for now. I’ve started looking into it and proper fix will require quite a lot of work, so it will take some time. Main issues:

• sqlite column type !== cell value type - this makes sqlite provider quite special as other databases we support doesn’t allow it
• system.data.sqlite provider has issues with type affinity handling (you use microsoft provider, but we cannot just ignore it), which could be solved, but it will break your case
• we don’t have data reader expression customization per-column data type right now, which is needed to unblock previous issue

[CumpsD]

We are storing them as ticks, and have the same wrapping issue.