[BUG] ApplicationTokenProvider should reuse HttpClientHandler instead of creating it on every call

Library name and version

Microsoft.Rest.ClientRuntime.Azure.Authentication 2.4.1

Describe the bug

Problem

Every call to ApplicationTokenProvider.LoginSilentAsync creates a new HttpClientHandler that will only be used only for the duration of the call.

This can easily lead to socket/port exhaustion.

The issue frequently creating and disposing HttpClientHandler explained in detail in the official Microsoft docs They mention HttpClient, but essentially the issue is with the HttpClientHandler that is created in HttpClient default constructor

Though this class implements IDisposable, declaring and instantiating it within a using statement is not preferred because when the HttpClient object gets disposed of, the underlying socket is not immediately released, which can lead to a socket exhaustion problem. For more information about this issue, see the blog post You’re using HttpClient wrong and it’s destabilizing your software.

After HttpClientHandler is disposed, the underlaying socket connections are still active for some time.

This issue caused (or contributed to) socket exhaustion on one of our production instances. I also found a related unanswered stackoverlfow issue describing exactly that problem.

Proposed solution

Most of the other azure sdk libraries provide a way to pass your own HttpClient. In our case, we use the default HttpClientFactory from ASP.NET that managers the lifetime of HttpClientHandler and clients.

There should be a way to pass HttpClient or HttpManager or something like that to ApplicationTokenProvider.LoginSilentAsync call.

The currently referenced version of Microsoft.IdentityModel.Clients.ActiveDirectory (4.3.0) does not support using IHttpClientFactory, the creation of HttpClientHandler is hard coded

However, an option to pass IHttpClientFactory was introduced from version 5.0.1-preview

Here’s a link to the instructions on how to use IHttpClientFactory with ADAL:

IHttpClientFactory myHttpClientFactory = new MyHttpClientFactory();

AuthenticationContext authenticationContext = new AuthenticationContext(
     authority: "https://login.microsoftonline.com/common",
     validateAuthority: true,
     tokenCache: <some token cache>,
     httpClientFactory: myHttpClientFactory);

That probably means changes to the following code and its callers to be able to pass IHttpClientFactory. https://github.com/Azure/azure-sdk-for-net/blob/f9858eac647b7f48bf63c8fb87defa656e759147/sdk/mgmtcommon/Auth/Az.Auth/Az.Authentication/ApplicationTokenProvider.cs#L678-L687

Workaround

I think I can replicate the behaviour of this ApplicationTokenProvider.LoginSilentAsync overload in our codebase to be able to pass HttpClientFactory to AuthenticationContext, but it should be fixed at the package level.

Expected behavior

It should be possible to pass HttpClient to ApplicationTokenProvider.LoginSilentAsync or to configure ApplicationTokenProvider to reuse HttpClientHandler to prevent socket exhaustion

Actual behavior

It should not possible to pass HttpClient to ApplicationTokenProvider.LoginSilentAsync or to configure ApplicationTokenProvider to reuse HttpClientHandler to prevent socket exhaustion. And it caused (or contributed to) a socket exhaustion in our production instance

Reproduction Steps

Here’s an example Program.cs that runs LoginSilentAsync in a loop and reports the count of active Tcp connections before and after each call. Please note, the count always increases, the connections are not even released on application stop.

Program.cs

using System;
using System.Threading.Tasks;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest.Azure.Authentication;

namespace HttpClientTestingTest
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var clientCredential = new ClientCredential("123", "345");

            for (var i = 0; i < 10; i++)
            {
                try
                {
                    ReportConnectionCount($"Before {i}");
                    await ApplicationTokenProvider.LoginSilentAsync("http://localhost/test", clientCredential, ActiveDirectoryServiceSettings.Azure);
                }
                catch
                {
                    ReportConnectionCount($"After {i}");
                }
            }
        }

        static void ReportConnectionCount(string info)
        {
            var properties = System.Net.NetworkInformation.IPGlobalProperties.GetIPGlobalProperties();
            Console.WriteLine($"ConnectionCount: {properties.GetActiveTcpConnections().Length} - {info}");
        }
    }
}

Environment

The production issue was happening in Azure App Service The reproduction on MacOS using netcore 3.1 and .NET 5.

[jsquire]

Thanks, @anaismiller, and apologies for the confusion. Assigning to @ArthurMa1978 for management library triage.

[jsquire]

Thank you for your feedback. Tagging and routing to the team members best able to assist.