Retrying in HttpMessageHandler hang for .NetFramework

Summary:

Create console application with .net framework 4.6.1. Add Polly 7.1.0 package.

Write following code (enable async main if not):

using Polly;
using System;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApp1
{
    class Program
    {
        static async Task Main(string[] args)
        {
            MyHandler h = new MyHandler();
            HttpClient c = new HttpClient(h);
            var result = await c.GetAsync("https://docs.microsoft.com/non-existing");
        }
    }

    class MyHandler : HttpClientHandler
    {
        protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            var policy = Policy.HandleResult<HttpResponseMessage>(message => !message.IsSuccessStatusCode).RetryAsync(
                6,
                (exception, retryCount, context) =>
                {
                    Console.WriteLine($"Retrying1: {retryCount}");
                    // do something
                });

            return policy.ExecuteAsync(async () => await base.SendAsync(request, cancellationToken));
        }
    }
}

Try run the code. The program hang after second retry. And the code works fine in .Net Core.

Main thread call stack:

mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext) Unknown mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) Unknown mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) Unknown mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.GetResult() Unknown ConsoleApp1.exe!ConsoleApp1.Program.<Main>(string[] args) Unknown

Working thread call stack:

mscorlib.dll!System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[] waitHandles, int millisecondsTimeout, bool exitContext) Unknown System.dll!System.Net.TimerThread.ThreadProc() Unknown mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) Unknown mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() Unknown

[reisenberger]

Thanks @jesong .

If you rewrite the same code with a manual retry loop (with no Polly) it also hangs in the same way. So the issue is outside Polly.

using System;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

namespace Issue642
{
    class Program
    {
        static async Task Main(string[] args)
        {
            MyHandler h = new MyHandler();
            HttpClient c = new HttpClient(h);
            var result = await c.GetAsync("https://docs.microsoft.com/non-existing");
        }
    }

    class MyHandler : HttpClientHandler
    {
        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            HttpResponseMessage result = null;
            for (int i = 0; i < 6; i++)
            {
                result = await base.SendAsync(request, cancellationToken);
                if (result.IsSuccessStatusCode) break;
                Console.WriteLine($"Retrying1: {i + 1}");
            }

            return result;
        }
    }
}

[jafz]

We hit this in production code, so I’m happy that this thread here existed. I had to explain why the issue was an issue. So I’ll post my try of an explanation of this issue: .NET limits the http client to 2 web requests by default. Why a connection is only disposed once the response is disposed might or might not be a bug.

Hence, this retry loop fails on the 3rd attempt, as 2 connections are already used. You could set this: HttpWebRequest.ServicePoint.ConnectionLimit to a very large number, or add this to the app.config:

  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="1000" />
    </connectionManagement>
  </system.net>

And then the reproduction example above will just send 6 requests. But I do not think that that’ll keep your code nice.

So, not only does dispose work around this issue, reading the content would do the same:

            for (int i = 0; i < 6; i++)
            {
                result = await base.SendAsync(request, cancellationToken);
                // reading the content has the same effect as disposing the result/response
                await result.Content.ReadAsStringAsync();
                // result.Dispose();
                if (result.IsSuccessStatusCode) break;
                Console.WriteLine($"Retrying1: {i + 1}");
            }

see: http://arnosoftwaredev.blogspot.com/2006/09/net-20-httpwebrequestkeepalive-and.html see: http://www.faqs.org/rfcs/rfc2616.html (8.1.4)