StorageExtendedErrorInformation.ReadFromStream causing deadlocks in .net core asp.net application

So, let me preface this by saying this is the first time I have ever used WinDBG and I’m kind of flying by the pants with this.

Our asp.net core application in production writes to blob storage (via appendblobs) every request for logging. We log specially formatted logs containing information about each request, parameters (with passwords redacted), exceptions, user information, etc…

The asp.net core application can go for over a week without issues and suddenly one instance on Azure will start hanging requests and returning 502s. Looking at the process details the dotnet.exe process is up to 700+ threads.

I took memory dumps via procmon, loaded sos, and ran clrstack on all the threads. Every thread I looked all had the exact same stack trace:

2d67e5f4 7728081c [GCFrame: 2d67e5f4] 
2d67e690 7728081c [HelperMethodFrame_1OBJ: 2d67e690] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
2d67e710 72233253 System.Threading.Monitor.Wait(System.Object, Int32, Boolean)
2d67e720 7223326c System.Threading.Monitor.Wait(System.Object, Int32)
2d67e724 722eadf3 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
2d67e778 722ba6dc System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
2d67e7b8 722ba5a9 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
2d67e814 722f8ba6 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
2d67e820 724550c9 System.Runtime.CompilerServices.TaskAwaiter`1[[System.Int32, System.Private.CoreLib]].GetResult()
2d67e828 6fc85890 System.Net.Http.WinHttpResponseStream.Read(Byte[], Int32, Int32)
2d67e844 6fc78c80 System.Net.Http.DelegatingStream.Read(Byte[], Int32, Int32)
2d67e854 6fe7547d System.Xml.XmlTextReaderImpl.InitStreamInput(System.Uri, System.String, System.IO.Stream, Byte[], Int32, System.Text.Encoding)
2d67e884 6fe73034 System.Xml.XmlTextReaderImpl.FinishInitStream()
2d67e894 6fe72fed System.Xml.XmlTextReaderImpl..ctor(System.IO.Stream, Byte[], Int32, System.Xml.XmlReaderSettings, System.Uri, System.String, System.Xml.XmlParserContext, Boolean)
2d67e8c4 6fe6f8c7 System.Xml.XmlReaderSettings.CreateReader(System.IO.Stream, System.Uri, System.String, System.Xml.XmlParserContext)
2d67e8e4 6fe6ebf6 System.Xml.XmlReader.Create(System.IO.Stream, System.Xml.XmlReaderSettings, System.String)
2d67e8f8 10909284 Microsoft.WindowsAzure.Storage.StorageExtendedErrorInformation.ReadFromStream(System.IO.Stream)
2d67e924 10908cb6 Microsoft.WindowsAzure.Storage.Core.Util.Exceptions+d__0.MoveNext()
2d67e984 1090882c System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Microsoft.WindowsAzure.Storage.Core.Util.Exceptions+d__0, Microsoft.WindowsAzure.Storage]](d__0 ByRef)
2d67e9b8 109087c0 Microsoft.WindowsAzure.Storage.Core.Util.Exceptions.PopulateStorageExceptionFromHttpResponseMessage(System.Net.Http.HttpResponseMessage, Microsoft.WindowsAzure.Storage.RequestResult, System.Func`4)
2d67ea04 10900dcb Microsoft.WindowsAzure.Storage.Core.Executor.Executor+d__4`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
2d67ecec 723143d1 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
2d67ecf4 722770eb System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2d67ed28 7231442b System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunnerWithContext.RunWithCapturedContext()
2d67ed4c 722ec367 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
2d67ed78 722bac20 System.Threading.Tasks.Task.FinishContinuations()
2d67ede8 722b972f System.Threading.Tasks.Task.FinishStageThree()
2d67edf4 722ef715 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon)
2d67ee04 723c9505 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetResult(System.__Canon)
2d67ee20 6fc97be4 System.Net.Http.HttpClient+d__58.MoveNext()
2d67ee74 723143d1 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
2d67ee7c 722770eb System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2d67eeb0 7231442b System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunnerWithContext.RunWithCapturedContext()
2d67eed4 722ec367 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
2d67ef00 722bac20 System.Threading.Tasks.Task.FinishContinuations()
2d67ef70 722b972f System.Threading.Tasks.Task.FinishStageThree()
2d67ef7c 722ef715 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon)
2d67ef8c 72403fcd System.Threading.Tasks.TaskCompletionSource`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon)
2d67ef9c 6fc9a7b8 System.Net.Http.WinHttpHandler+d__105.MoveNext()
2d67f050 723143d1 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
2d67f058 722770eb System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2d67f08c 7231442b System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunnerWithContext.RunWithCapturedContext()
2d67f0b0 722ec1ac System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
2d67f0bc 7231265d System.Threading.ThreadPoolWorkQueue.Dispatch()
2d67f10c 72404b5a System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
2d67f31c 72bf309f [DebuggerU2MCatchHandlerFrame: 2d67f31c] 

From what I can tell, the cause seems to be that StorageExtendedErrorInfomration.ReadFromStream() calls XmlReader which is not asynchronous. XmlReader calls https://github.com/dotnet/corefx/blob/master/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseStream.cs#L248 which wraps an asynchronous call in a Task and blocks via GetResult(). This seems to be the root cause of our deadlock.

I assume we are just getting transient error spikes from Azure blob storage, which is causing all these logging requests to error at the same time. This causes the thread pool to go over the 682 limit (verified via !ThreadPool in windbg) and the whole process completely deadlocks in a way that can’t be easily auto-mitigated or healed against.

On our end the only thing I can do from here is to rip out azure storage and go to some other mechanism.

[erezvani1529]

Hi @KallDrexx,

As you have correctly identified the synchronous read call from XmlReader triggers the sync over async call causing the thread exhaustion issue. We have been internally following up with CoreFx networking team since we discovered this pattern and will post upon any updates we receive.

However, we will have a fix/workaround on our end in the error parsing path until we make the refactoring change we need, to fully support async reads from response. We are planning to include this change in our upcoming release 8.2 as a bug fix.

This issue will be used to track the change.

Thanks, Elham

[erezvani1529]

This issue has been in v8.2+. Please let us know if you see any further issues.

Thanks!