EF Core 2.0 Possible regression when filtering results

Code that works fine in EF Core1.1.5 fails using EF Core 2.0.2 (same in 2.1 preview). We are working on a 40+ projects solution with a lot of repositories.

Migrating from EF Core 1.1.x to 2.0.x we have found that a lot of our queries now are failing with a NullReferenceException, or missing data on our endpoints, due to Navigation properties not materialized when using a filter (Where clause) that cannot be translated in SQL, so executed locally.

Not sure if it is a bug, but I didn’t find anything in the release notes related to this behavior change.

I add a SLN example with the exact same code running EF Core 1 or 2 EF1vsEF2.zip

The example is using the SQLite provider running an InMemory DB, but we use Azure SQL Databases in our project…with the same outcome.

Here is the Program.cs file (all needs self contained):

using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;

namespace EF1
{
    static class Program
    {
        class Entity
        {
            [Key]
            public int EntityID { get; set; }

            public int StatusID { get; set; }

            [ForeignKey("StatusID")]
            [InverseProperty("Entities")]
            public virtual Status Status { get; set; }
        }

        class Status
        {
            [Key]
            public int StatusID { get; set; }

            [Required]
            public string Code { get; set; }


            [InverseProperty("Status")]
            public ICollection<Entity> Entities { get; set; }

            public Status()
            {
                Entities = new HashSet<Entity>();
            }
        }

        class AppDbContext : DbContext
        {
            public DbSet<Entity> Entities { get; set; }
            public DbSet<Status> Statuses { get; set; }
            public AppDbContext(DbContextOptions options) : base(options)
            {
            }
        }

        static async Task Main(string[] args)
        {
            var connection = new SqliteConnection("DataSource=:memory:");
            connection.Open();

            var options = new DbContextOptionsBuilder<AppDbContext>()
                .UseSqlite(connection)
                .Options;

            try
            {
                //Seed DB
                Console.ForegroundColor = ConsoleColor.Cyan;
                Console.WriteLine("EF CORE 1.1.5 Test");

                using (var ctx = new AppDbContext(options))
                {
                    ctx.Database.EnsureCreated();

                    var status = new Status { Code = "CODE VALUE" };
                    var entity = new Entity { Status = status };

                    ctx.Add(entity);
                    ctx.SaveChanges();
                }

                //Fetch data
                using (var ctx = new AppDbContext(options))
                {
                    var entities = await ctx.Entities
                        .Include(e => e.Status)
                        .Where(e => e.FakeFilterNotRenderedInSQL())
                        .ToListAsync();

                    foreach (var entity in entities)
                    {
                        Console.ForegroundColor = entity.Status == null ? ConsoleColor.Red : ConsoleColor.Green;
                        Console.WriteLine($"After Materialization -> Entity has ID: {entity.EntityID}, Status code: {entity.Status?.Code ?? "Not Materialized"}");
                        Console.ForegroundColor = ConsoleColor.White;
                    }
                }
            }
            finally
            {
                connection.Close();
            }

            Console.ReadKey();
            
        }

        static bool FakeFilterNotRenderedInSQL(this Entity entity)
        {
            Console.ForegroundColor = entity.Status == null ? ConsoleColor.Red : ConsoleColor.Green;
            Console.WriteLine($"Inside IQueryable<Entity>.Where clause -> Entity has ID: {entity.EntityID}, Status code: {entity.Status?.Code ?? "Not Materialized"}");
            Console.ForegroundColor = ConsoleColor.White;
            return true;
        }
    }
}

Here is the outputs:

Our migration is in standby because identifying all code that could be impacted by that regression will need a significat amount of time and tests.

A workaround is to materialize the entities before filtering, but it is not a satisfying solution regarding all code needed to be modified.

Best regards.

[maumar]

Correlated collection + include unification could help this scenario - problem currently is that we generate query plan for include before we know that client evaluation is needed. Unification will push that to VisitQueryModel phase, at which point we already know that predicate needs client evaluation and can come up with a different execution plan.

[divega]

@maumar please add the reference to this scenario in the Include unification issue. Then you can close as duplicate.