AsSplitQuery sporadically missing navigation properties

This is not going to be much of a bug report because I’m currently unable to reliably reproduce the issue described but I wanted to maybe get a conversation started that might point me in the right direction.

We use the EF Core 5.0.8 against MS SQL Server 2016 setup in a synchronized cluster.

The situation is as follows: we have a table which for each “entity” contains two rows - a baseline and a modified version. Since the two use the same schema, they are in the same table. There is no self-referencing key binding the original to the modified item.

We had a performance issue with a specific query (the table has a lot of columns + several child collections) so we decided to enable query splitting. Since then, we sporadically (< 5% of executions) get into a situation where one of the child collections fails to load, leading to data loss down the line.

As I said, I’m currently unable to replicate this anywhere except our production environment. The query in question looks like this:

var original = _context.VehicleInspections.Filter(...)
                .Include(i => i.Motors) // collection
                .Include(i => i.Vehicle) // single
                .Include(i => i.Obligations) // collection
                .Include(i => i.ServiceItems) // collection
                .Where(i => i.RecordType == InspectionRecordType.Original);

var modified = _context.VehicleInspections.Filter(...)
                .Where(i => i.RecordType == InspectionRecordType.Modified)
                .Include(i => i.Motors)
                .Include(i => i.Vehicle)
                .Include(i => i.Obligations)
                .Include(i => i.ServiceItems)
                .Where(i => i.State == InspectionState.Closed)
                .Where(i => !i.Acknowledged);

var leftJoin = from mod in modified
                join orig in original on new { mod.CaseId, mod.SequenceNumber } equals new { orig.CaseId, orig.SequenceNumber }
                    into joined
                from o in joined.DefaultIfEmpty()
                select new InspectionPair { Original = o, Modified = mod };
var inspections = await leftJoin.ToArrayAsync(cancellationToken);

This produces a left self-join over the table in question. The query split is performed correctly and the subqueries look reasonable to my eye. However, still the Obligations collection on the Modified entity sometimes remains empty even though we know there are child entities present. Without query splitting, this works as expected via a single large query.

Since I cannot easily produce a repro code, I guess I just wanted to task - is there anything suspicious in this code that could cause such behavior? I noticed there are some navigation expansion bugs being tracked for 5.x but this issue seems to be pop up non-deterministically - most executions still work as expected (or at least we haven’t been able to isolate the cause yet).

[mareklinka]

Tomorrow I will attempt to determine whether the SQL Server actually delivers the “missing” data to EF. I wrote a DbCommandInterceptor that reads the content of the data reader and logs the IDs of all the obligations within. This should, hopefully, confirm whether the problem lies with EF or the SQL Server.

[AppleRyan]

Met with the same problem in our prod environments when using splitting query.

This is not going to be much of a bug report because I’m currently unable to reliably reproduce the issue described but I wanted to maybe get a conversation started that might point me in the right direction.

We use the EF Core 5.0.8 against MS SQL Server 2016 setup in a synchronized cluster.

The situation is as follows: we have a table which for each “entity” contains two rows - a baseline and a modified version. Since the two use the same schema, they are in the same table. There is no self-referencing key binding the original to the modified item.

We had a performance issue with a specific query (the table has a lot of columns + several child collections) so we decided to enable query splitting. Since then, we sporadically (< 5% of executions) get into a situation where one of the child collections fails to load, leading to data loss down the line.

As I said, I’m currently unable to replicate this anywhere except our production environment. The query in question looks like this:

var original = _context.VehicleInspections.Filter(...)
                .Include(i => i.Motors) // collection
                .Include(i => i.Vehicle) // single
                .Include(i => i.Obligations) // collection
                .Include(i => i.ServiceItems) // collection
                .Where(i => i.RecordType == InspectionRecordType.Original);

var modified = _context.VehicleInspections.Filter(...)
                .Where(i => i.RecordType == InspectionRecordType.Modified)
                .Include(i => i.Motors)
                .Include(i => i.Vehicle)
                .Include(i => i.Obligations)
                .Include(i => i.ServiceItems)
                .Where(i => i.State == InspectionState.Closed)
                .Where(i => !i.Acknowledged);

var leftJoin = from mod in modified
                join orig in original on new { mod.CaseId, mod.SequenceNumber } equals new { orig.CaseId, orig.SequenceNumber }
                    into joined
                from o in joined.DefaultIfEmpty()
                select new InspectionPair { Original = o, Modified = mod };
var inspections = await leftJoin.ToArrayAsync(cancellationToken);

This produces a left self-join over the table in question. The query split is performed correctly and the subqueries look reasonable to my eye. However, still the Obligations collection on the Modified entity sometimes remains empty even though we know there are child entities present. Without query splitting, this works as expected via a single large query.

Since I cannot easily produce a repro code, I guess I just wanted to task - is there anything suspicious in this code that could cause such behavior? I noticed there are some navigation expansion bugs being tracked for 5.x but this issue seems to be pop up non-deterministically - most executions still work as expected (or at least we haven’t been able to isolate the cause yet).