Connections from the pool are not reused when using async methods in parallel

Moved from https://github.com/aspnet/EntityFrameworkCore/issues/10169 reported by @wertzui

Whem I’m using async methods (like FirstOrDefaultAsync) to get a result from the Database inside a Parallel.For, the connections are not correctly reused from the connection pool.

The connections will rise to the configured Max Pool Size and after some time (around 1-2 minutes on my machine using localdb), exceptions will be thrown.

Using either async methods and a normal for-loop or non-async methods an a Parallel.For-loop, the connections are properly reused and I can observe that a normal for-loop uses 1 connection and a Parallel.For-loop uses 4 connections which corresponds to MaxDegreeOfParallelism.

Exception message:
System.InvalidOperationException occurred
  HResult=0x80131509
  Message=Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
Stack trace:
   at System.Data.Common.ADP.ExceptionWithStackTrace(Exception e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.<OpenAsync>d__31.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable.AsyncEnumerator.<BufferlessMoveNext>d__9.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<ExecuteImplementationAsync>d__33`2.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<ExecuteImplementationAsync>d__33`2.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable.AsyncEnumerator.<MoveNext>d__8.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.SelectAsyncEnumerable`2.SelectAsyncEnumerator.<MoveNext>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.<_FirstOrDefault>d__82`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Query.Internal.TaskResultAsyncEnumerable`1.Enumerator.<MoveNext>d__3.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.SelectAsyncEnumerable`2.SelectAsyncEnumerator.<MoveNext>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.<MoveNext>d__5.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at EfPoolingTest.UnitTest1.<>c.<NormalContextParallel>b__4_0(Int32 i) in C:\Users\xxx\documents\visual studio 2017\Projects\EfPoolingTest\EfPoolingTest\UnitTest1.cs:line 70
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)

Steps to reproduce

• Clone https://github.com/wertzui/EfPoolingTest and open the project
• Run the “ViewConnections.sql” script to see that there are no connections open (beside the 2 used for the script)
• Run the test “NormalContextParallelAsyncConfigureAwaitFalse” (I suggest running in Debug mode, so you can see the output)
• Run the “ViewConnections.sql” script multiple times to see the connections increase.
• After some time you will also get an exception
• Run the other tests and observe the connection count

Further technical details

EF Core version: 2.0.0 (also tested with 1.1.3) Database Provider: Microsoft.EntityFrameworkCore.SqlServer/localdb Operating system: Windows 10 FCU IDE: Visual Studio 2017 15.4.1

[Wraith2]

There isn’t a fault in the SqlClient, it’s releasing the connections back into the pool as soon as it’s able to. What is happening is thread starvation.

The parallel operation attempt to maintain a number of operations running at the same time. Each of those operations does a async to sync transition using GetAwaiter().GetResult() and in doing so causes the thread making that call to block. So you pick up a thread from the threadpool and then use it to issue a wait on a ManualResetEventSlim which needs to be signalled from another thread, that other thread is also from the threadpool. Eventually you end up with all threadpool threads being busy waiting and no more threads be available to scheduler the completers. The threadpool detects the load and will slowly add more threads and each time it adds some threads they’ll run for a while possibly unblocking some previous actions until the system once again reaches a steady state of being blocked. In the extreme long term the threadpool thread count will exceed the number of sql pooled connections and you’ll end up unable to get a connection to work with.

Graphically in the profiler you see this: Where some blue patches of work are done and then everything ends up in grey wait state. Then a new thread gets spawned which completed some work allowing other threads to unblock and do a little work and then it all settled back into waiting again quickly. You can see that the new threads are added every second or so in this example which is the threadpool throttling new thread creation as it’s supposed to.

In those grey waiting periods the call stacks are like this: Everything is just waiting for async work to complete by using a MRES blocking wait.

So how do you fix it? Glib answer: don’t do this. Conceptual answer don’t transition from sync to async, the worlds don’t mix nicely but this might not be practical. Practically, don’t use parallel sync waits to implement this behaviour. How you work around this depends entirely on the features of your problem. You could manually batch up async tasks and then do a Task.WaitAll(taskArray).GetAwaiter().GetResult() in a loop until you’re done with however much input you’ve got to handle. use the GetAwaiter().GetResult() calls as little as possible and as high up in the call chain as you can. If your input is small you could just start a new detached task for each input and then wait on them all to finish with Task.WaitAll.

My question is why would you do this in the first place? You’ve got a set of input values that you want to operate on in parallel. That’s exactly what SQL server is good at. Why not feed the whole lot of input into a stored procedure as a range or a tvp list and then issue a single command to do them all.

[cheenamalhotra]

Hi @eglauko

It was in my list of things but I was wondering if fixing for 2.2 would be needed as .NET Core 2.2 is reaching End of Life in December (a couple of weeks left).

Is it possible to upgrade System.Data.SqlClient to v4.7.0 by specifying explicit package reference and continue using with your application?

System.Data.SqlClient v4.7.0 contains fix for this issue.