Disposing an EF enumerator takes long time on large/complex queries on SQL Server.

When you have a query that returns lots of records, the disposing of an enumerator takes long time if you aborts the enumeration before traversing through all rows.

I did a simple test project that has a simple table with 2 million rows and executed the following code. The metrics for the disposing was approx. 1 second on my environment when creating a data reader without filtering (other filtering is also done in the test project.

foreach (var t in context.Tests)
{
    break; //Just to stop the iteration directly
} // This takes approx one second when it disposes the enumerator foreach uses

After investigating the EF core code around this, I see that the Dispose method in RelationalDataReader.cs calls Close on the data reader. According to the Remarks section of the SqlDataReader.Close method it states:

The Close method fills in the values for output parameters, return values and RecordsAffected, increasing the time that it takes to close a SqlDataReader that was used to process a large or complex query.

And this is the problem; if you haven’t iterated through the reader to the end, the close method will iterate through the rest of the reader and thus create a heavy performance impact against SQL Server on large/complex queries.

Further, the remarks section follows up with

When the return values and the number of records affected by a query are not significant, the time that it takes to close the SqlDataReader can be reduced by calling the Cancel method of the associated SqlCommand object before calling the Close method.

The number of rows affected isn’t interesting in when iterating, so the disposing logic should call Cancel on the SqlDataReader. I see that this may be an issue to solve this easily in the code as it is, since the RelationalDataReader uses the DbDataReader which doesn’t have an Cancel logic… But again, it is something that heavily impacts performance when certain operations are done, so it should be fixed some way.

This issue has been reproduced with the following environment:

.NET SDK (reflecting any global.json):
 Version:   5.0.201
 Commit:    a09bd5c86c

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19042
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.201\

Host (useful for support):
  Version: 5.0.4
  Commit:  f27d337295

.NET SDKs installed:
  3.1.301 [C:\Program Files\dotnet\sdk]
  3.1.407 [C:\Program Files\dotnet\sdk]
  5.0.104 [C:\Program Files\dotnet\sdk]
  5.0.201 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

And with Microsoft.EntityFrameworkCore.SqlServer Version 5.0.5

[roji]

When a reader is disposed (e.g. because an EF Enumerator is disposed), any pending results must indeed be consumed - this is necessary in order for the database connection to be usable for another query. In your example app, you execute a query which fetches a huge number of rows, and then immediately dispose the reader; it’s expected for this to be quite slow (the notes on the number of records affected as well as output/return parameters are very unlikely to be relevant for this case).

Systematically calling DbCommand.Cancel on unconsumed readers would most probably not be a good thing, as it would likely negatively impact queries where there isn’t a huge number of pending rows. However, you should be able to trigger this behavior from user code by executing your query asynchronously, and then triggering the cancellation token before disposing the enumerator.

However, at the end of the day, it’s usually bad practice to be selecting rows which you won’t be consuming - it’s recommended to use the Take operator (and paging in general) to fetch rows which you know you’ll need. There are indeed some cases where the number of rows needed isn’t known until you start enumerating the resultset, but these cases should be rare, and it may still be better to simply request more rows using paging (at the cost of additional roundtrips).

[AndriySvyryd]

EF 5.0 does have a bug, where RelationalDataReader.DisposeAsync (async) calls DbDataReader.Close (sync). This was already fixed for 6.0 as part of #24207. We can consider patching this.

Unless we get more reports we don’t think there’s enough value in patching this in 5.0.x