Microsoft.Data.SqlClient 3.0.0 breaks async enumeration of results of SQL Server query including null rowversion value

Upgrading to Microsoft.Data.SqlClient 3.0.0 results in InvalidCastException (“Unable to cast object of type ‘System.DBNull’ to type ‘System.Byte[]’”) - that does not occur with Microsoft.Data.SqlClient 2.1.3 - when async enumerating over the results of a query that includes null rowversion values and when sqlOptions.EnableRetryOnFailure().

I thought this might be something to do with https://github.com/dotnet/SqlClient/pull/998. However, enabling the LegacyRowVersionNullBehaviour switch does not fix the problem.

In trying to narrow down a repro, it became clear the error only occurs if sqlOptions.EnableRetryOnFailure() is called when configuring the context. This, plus the fact that non-async enumeration of the same query works seems to suggest problem in EfCore.

Versions

Observed with:

• 5.0.7
• 6.0.0-preview.4.21253.1

Repro:

Repro project at: https://github.com/frankbuckley/efcore-sqldata3

Database:

drop table if exists dbo.Price;
go

drop table if exists dbo.Occurrence;
go

create table dbo.Occurrence
(
    Id        int          not null identity,
    Title     nvarchar(80) not null,
    Timestamp rowversion   not null,
    constraint pk_Occurrence
        primary key clustered (Id)
);
create table dbo.Price
(
    OccurrenceId int        not null,
    Currency     char(3)    not null,
    Value        decimal    not null,
    Timestamp    rowversion not null,
    constraint pk_Price
        primary key clustered (OccurrenceId, Currency),
    constraint fk_Price_Occurrence
        foreign key (OccurrenceId)
        references dbo.Occurrence (Id)
);
go

Program:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;

namespace EfCoreMsSqlData3
{
    internal class Program
    {
        private static async Task Main(string[] args)
        {
            // Makes no difference

            // AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.LegacyRowVersionNullBehaviour", true);

            using (EventsDbContext db = new())
            {
                if ((await db.Occurrences.CountAsync()) == 0)
                {
                    // Note: no prices, therefore LEFT JOIN when included in query of occurrences will return nulls

                    for (int i = 0; i < 10; i++)
                    {
                        db.Occurrences.Add(new Occurrence { Title = "Test " + i });
                    }

                    await db.SaveChangesAsync();
                }
            }

            // This works

            using (EventsDbContext db = new())
            {
                foreach (Occurrence? o in db.Occurrences.Include(o => o.Prices))
                {
                    Console.WriteLine(o.Title + " (" + o.Timestamp + ")");
                }
            }

            // This fails

            using (EventsDbContext db = new())
            {
                await foreach (Occurrence? o in db.Occurrences.Include(o => o.Prices).AsAsyncEnumerable())
                {
                    Console.WriteLine(o.Title + " (" + o.Timestamp + ")");
                }
            }
        }
    }

    public class EventsDbContext : DbContext
    {
        private const string Connection = "Data Source=(local);Initial Catalog=EfCoreMsSqlData3;" +
            "Integrated Security=True;Connect Timeout=60;Encrypt=False;TrustServerCertificate=False;" +
            "ApplicationIntent=ReadWrite;MultiSubnetFailover=False";

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder
                .EnableDetailedErrors()
                .EnableSensitiveDataLogging()
                .UseSqlServer(Connection, options =>
                {
                    // Remove this and it works...

                    options.EnableRetryOnFailure();
                })
                .LogTo(Console.WriteLine);
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<Occurrence>()
                .ToTable("Occurrence")
                .HasKey(o => o.Id);

            modelBuilder.Entity<Occurrence>()
                .Property(o => o.Timestamp)
                .IsRowVersion();

            modelBuilder.Entity<Occurrence>()
                .HasMany(o => o.Prices)
                .WithOne(o => o.Occurrence)
                .HasForeignKey(p => p.OccurrenceId);

            modelBuilder.Entity<Price>()
                .ToTable("Price")
                .HasKey(p => new { p.OccurrenceId, p.Currency });

            modelBuilder.Entity<Price>()
                .Property(o => o.Timestamp)
                .IsRowVersion();
        }

        public DbSet<Occurrence> Occurrences { get; set; }
    }


    public abstract class PersistedObject
    {
        public byte[] Timestamp { get; set; }
    }

    public abstract class Entity<TId> : PersistedObject
        where TId : IEquatable<TId>
    {
        public TId Id { get; set; }
    }

    public class Occurrence : Entity<int>
    {
        public string Title { get; set; }

        public List<Price> Prices { get; set; }
    }

    public class Price : PersistedObject
    {
        public int OccurrenceId { get; set; }

        public string Currency { get; set; }

        public Occurrence Occurrence { get; set; }

        public decimal Value { get; set; }
    }
}

Stacktrace:

System.InvalidOperationException: An error occurred while reading a database value for property 'Price.Timestamp'. The expected type was 'System.Byte[]' but the actual value was null.
       ---> System.InvalidCastException: Unable to cast object of type 'System.DBNull' to type 'System.Byte[]'.
         at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueFromSqlBufferInternal[T](SqlBuffer data, _SqlMetaData metaData)
         at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueInternal[T](Int32 i)
         at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValue[T](Int32 i)
         at lambda_method58(Closure , DbDataReader , Int32[] )
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadObject(DbDataReader reader, Int32 ordinal, ReaderColumn column)
         --- End of inner exception stack trace ---
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadObject(DbDataReader reader, Int32 ordinal, ReaderColumn column)
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadRow()
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

Environment

Originally discovered in integration tests running on Ubuntu 20.04 with SDK 5.0.301 and Azure SQL Database.

Repro tested on Windows with local SQL Server 15.0.2080.9:

dotnet --info

.NET SDK (reflecting any global.json):
 Version:   5.0.301
 Commit:    ef17233f86

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19043
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.301\

Host (useful for support):
  Version: 5.0.7
  Commit:  556582d964

.NET SDKs installed:
  3.1.410 [C:\Program Files\dotnet\sdk]
  5.0.100 [C:\Program Files\dotnet\sdk]
  5.0.202 [C:\Program Files\dotnet\sdk]
  5.0.204 [C:\Program Files\dotnet\sdk]
  5.0.300 [C:\Program Files\dotnet\sdk]
  5.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[roji]

Checked with the just-released 4.0.0-preview1.21237.2, and the bug no longer repro’s there.

[roji]

I’ve tracked this down to what looks like a bug in SqlClient 3.0.0, opened https://github.com/dotnet/SqlClient/issues/1228 to track. In a nutshell, SqlDataReader.IsDbNull returns wrong results for null timestamp when used after ReadAsync.

Note also that The fix for LegacyRowVersionNullBehaviour has been merged for 4.0.0-preview1 (hopefully also to be backported to 3.0.1), though if https://github.com/dotnet/SqlClient/issues/1228 is fixed that AppContext switch shouldn’t be necessary for EF Core to work properly.