question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] ApplicationTokenProvider should reuse HttpClientHandler instead of creating it on every call

See original GitHub issue

Library name and version

Microsoft.Rest.ClientRuntime.Azure.Authentication 2.4.1

Describe the bug

Problem

Every call to ApplicationTokenProvider.LoginSilentAsync creates a new HttpClientHandler that will only be used only for the duration of the call.

This can easily lead to socket/port exhaustion.

The issue frequently creating and disposing HttpClientHandler explained in detail in the official Microsoft docs They mention HttpClient, but essentially the issue is with the HttpClientHandler that is created in HttpClient default constructor

Though this class implements IDisposable, declaring and instantiating it within a using statement is not preferred because when the HttpClient object gets disposed of, the underlying socket is not immediately released, which can lead to a socket exhaustion problem. For more information about this issue, see the blog post You’re using HttpClient wrong and it’s destabilizing your software.

After HttpClientHandler is disposed, the underlaying socket connections are still active for some time.

This issue caused (or contributed to) socket exhaustion on one of our production instances. I also found a related unanswered stackoverlfow issue describing exactly that problem.

Proposed solution

Most of the other azure sdk libraries provide a way to pass your own HttpClient. In our case, we use the default HttpClientFactory from ASP.NET that managers the lifetime of HttpClientHandler and clients.

There should be a way to pass HttpClient or HttpManager or something like that to ApplicationTokenProvider.LoginSilentAsync call.

The currently referenced version of Microsoft.IdentityModel.Clients.ActiveDirectory (4.3.0) does not support using IHttpClientFactory, the creation of HttpClientHandler is hard coded

However, an option to pass IHttpClientFactory was introduced from version 5.0.1-preview

Here’s a link to the instructions on how to use IHttpClientFactory with ADAL:

IHttpClientFactory myHttpClientFactory = new MyHttpClientFactory();

AuthenticationContext authenticationContext = new AuthenticationContext(
     authority: "https://login.microsoftonline.com/common",
     validateAuthority: true,
     tokenCache: <some token cache>,
     httpClientFactory: myHttpClientFactory);

That probably means changes to the following code and its callers to be able to pass IHttpClientFactory. https://github.com/Azure/azure-sdk-for-net/blob/f9858eac647b7f48bf63c8fb87defa656e759147/sdk/mgmtcommon/Auth/Az.Auth/Az.Authentication/ApplicationTokenProvider.cs#L678-L687

Workaround

I think I can replicate the behaviour of this ApplicationTokenProvider.LoginSilentAsync overload in our codebase to be able to pass HttpClientFactory to AuthenticationContext, but it should be fixed at the package level.

Expected behavior

It should be possible to pass HttpClient to ApplicationTokenProvider.LoginSilentAsync or to configure ApplicationTokenProvider to reuse HttpClientHandler to prevent socket exhaustion

Actual behavior

It should not possible to pass HttpClient to ApplicationTokenProvider.LoginSilentAsync or to configure ApplicationTokenProvider to reuse HttpClientHandler to prevent socket exhaustion. And it caused (or contributed to) a socket exhaustion in our production instance

Reproduction Steps

Here’s an example Program.cs that runs LoginSilentAsync in a loop and reports the count of active Tcp connections before and after each call. Please note, the count always increases, the connections are not even released on application stop.

Program.cs
using System;
using System.Threading.Tasks;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest.Azure.Authentication;

namespace HttpClientTestingTest
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var clientCredential = new ClientCredential("123", "345");

            for (var i = 0; i < 10; i++)
            {
                try
                {
                    ReportConnectionCount($"Before {i}");
                    await ApplicationTokenProvider.LoginSilentAsync("http://localhost/test", clientCredential, ActiveDirectoryServiceSettings.Azure);
                }
                catch
                {
                    ReportConnectionCount($"After {i}");
                }
            }
        }

        static void ReportConnectionCount(string info)
        {
            var properties = System.Net.NetworkInformation.IPGlobalProperties.GetIPGlobalProperties();
            Console.WriteLine($"ConnectionCount: {properties.GetActiveTcpConnections().Length} - {info}");
        }
    }
}

Environment

The production issue was happening in Azure App Service The reproduction on MacOS using netcore 3.1 and .NET 5.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
jsquirecommented, Feb 10, 2022

Thanks, @anaismiller, and apologies for the confusion. Assigning to @ArthurMa1978 for management library triage.

1reaction
jsquirecommented, Feb 9, 2022

Thank you for your feedback. Tagging and routing to the team members best able to assist.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Should I cache and reuse HttpClient created from ...
So, technically it doesn't matter whether you cache the HttpClient or dispose it straight away - disposing it doesn't do anything (because it's ......
Read more >
What's the recommended approach for reusing/disposing ...
It's been recommended to reuse the full framework's HttpClientHandler across multiple requests instead of disposing it every time (because ...
Read more >
You're using HttpClient wrong and it is destabilizing your ...
This means that under the covers it is reentrant and thread safe. Instead of creating a new instance of HttpClient for each execution...
Read more >
Should we create a new single instance of HttpClient for all ...
So what is probably happening when an HttpClient is shared is that the connections are being reused, which is fine if you don't...
Read more >
HttpClient guidelines for .NET
The factory pools HttpMessageHandler instances, and, if its lifetime hasn't expired, a handler can be reused from the pool when the factory  ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found