Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] KeyVaultClient keeps httpclient socket connections open

See original GitHub issue

Describe the bug We have a Appservice running which gets a value from keyvault each 10 minutes (we cache the value for 10 minutes). We host a appservice on azure. .net framework 4.6.1 Is it a know problem that the Azure.KeyVault client is not closing socket connections? I couldn’t found any related issues.

To Reproduce Steps to reproduce the behavior: We add the class below with a singleton interface. It is invoked each httprequest.

Instance	RemoteAddress	Established	TimeWait
[xxxx]	51.144.124.X:5671	4077	0

Code Snippet Add the code snippet that causes the issue.

public class KeyVaultUsernamePasswordChecker
    {
        private string _keyVaultSecretsBaseUri;
        private readonly AzureServiceTokenProvider _azureServiceTokenProvider;
        private readonly KeyVaultClient _keyVaultClient;
        private readonly ILogger<KeyVaultUsernamePasswordChecker> _logger;
        private readonly IMemoryCache _memoryCache;

        public KeyVaultUsernamePasswordChecker(IConfiguration configuration, ILogger<KeyVaultUsernamePasswordChecker> logger, IMemoryCache memoryCache)
        {
            _keyVaultSecretsBaseUri = configuration.GetValue<string>("AppSettings:KeyVaultSecretsBaseUri").TrimEnd('/');
            _azureServiceTokenProvider = new AzureServiceTokenProvider();
            _keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(_azureServiceTokenProvider.KeyVaultTokenCallback));
            _logger = logger;
            _memoryCache = memoryCache;
        }

        public async Task<bool> IsValidUserAsync(string username, string password)
        {
            try
            {
                var secretKey = string.Concat(_keyVaultSecretsBaseUri, "/", username);
                var cachedSecretValue = string.Empty;
                var found = _memoryCache.TryGetValue(secretKey, out cachedSecretValue);

                if (found && cachedSecretValue == password) return true;

                var secret = await _keyVaultClient.GetSecretAsync(secretKey);

                _memoryCache.Set(secretKey, secret.Value, TimeSpan.FromMinutes(10));

                var result = secret.Value == password;

                if (result) return true;

                _logger.LogWarning($"Password for user '{username}' is incorrect");

                return false;
            }
            catch (KeyVaultErrorException ex)
            {
                _logger.LogWarning($"User '{username}' not found in KeyVault ('{ex.Message}')");

                return false;
            }
            catch (Exception ex)
            {
                _logger.LogError(new EventId(), ex, $"Unhandled exception while connecting to KeyVault for user/password validation");

                return false;
            }
        }
    }

Expected behavior A clear and concise description of what you expected to happen. The httpclient is not closed properly so it socket connections are invoked each time

Setup (please complete the following information):

OS: Azure webapp
Microsoft Azure KeyVault 2.3.2 (because of breaking change we cannot upgrade to 3.x)

Exception or Stack Trace System.Net.Sockets.SocketException

[
	{
		"parsedStack": [
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
				"level": 0,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
				"level": 1,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "Microsoft.Rest.RetryDelegatingHandler+<>c__DisplayClass11_0+<<SendAsync>b__1>d.MoveNext",
				"level": 2,
				"line": 0,
				"assembly": "Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
				"level": 3,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
				"level": 4,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "Microsoft.Rest.RetryDelegatingHandler+<SendAsync>d__11.MoveNext",
				"level": 5,
				"line": 0,
				"assembly": "Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
				"level": 6,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
				"level": 7,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__58.MoveNext",
				"level": 8,
				"line": 0,
				"assembly": "System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
				"level": 9,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
				"level": 10,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd",
				"level": 11,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "Microsoft.Azure.KeyVault.KeyVaultClient+<GetSecretWithHttpMessagesAsync>d__65.MoveNext",
				"level": 12,
				"line": 0,
				"assembly": "Microsoft.Azure.KeyVault, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
				"level": 13,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
				"level": 14,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "Microsoft.Azure.KeyVault.KeyVaultClientExtensions+<GetSecretAsync>d__12.MoveNext",
				"level": 15,
				"line": 0,
				"assembly": "Microsoft.Azure.KeyVault, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
				"level": 16,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
				"level": 17,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
				"level": 18,
				"line": 0,
				"assembly": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"fileName": "KeyVaultUsernamePasswordChecker.cs",
				"method": "AppService.Security.KeyVaultUsernamePasswordChecker+<IsValidUserAsync>d__6.MoveNext",
				"level": 19,
				"line": 39,
				"assembly": "DataServices.Web, Version=1.14.0.0, Culture=neutral, PublicKeyToken=null"
			}
		],
		"outerId": "0",
		"message": "An error occurred while sending the request.",
		"type": "System.Net.Http.HttpRequestException",
		"id": "36241"
	},
	{
		"parsedStack": [
			{
				"method": "System.Net.HttpWebRequest.EndGetResponse",
				"level": 0,
				"line": 0,
				"assembly": "System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Net.Http.HttpClientHandler.GetResponseCallback",
				"level": 1,
				"line": 0,
				"assembly": "System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
			}
		],
		"outerId": "36241",
		"message": "Unable to connect to the remote server",
		"type": "System.Net.WebException",
		"id": "10038813"
	},
	{
		"parsedStack": [
			{
				"method": "System.Net.Sockets.Socket.DoBind",
				"level": 0,
				"line": 0,
				"assembly": "System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Net.Sockets.Socket.InternalBind",
				"level": 1,
				"line": 0,
				"assembly": "System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Net.Sockets.Socket.BeginConnectEx",
				"level": 2,
				"line": 0,
				"assembly": "System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Net.Sockets.Socket.UnsafeBeginConnect",
				"level": 3,
				"line": 0,
				"assembly": "System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			},
			{
				"method": "System.Net.ServicePoint.ConnectSocketInternal",
				"level": 4,
				"line": 0,
				"assembly": "System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
			}
		],
		"outerId": "10038813",
		"message": "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full",
		"type": "System.Net.Sockets.SocketException",
		"id": "29287854"
	}

Issue Analytics

State:
Created 4 years ago
Comments:8 (4 by maintainers)

Top GitHub Comments

1reaction

JonathanHusscommented, Jul 11, 2019

An update:

Is seems as though maybe the problem actually resides in the underlying AzureServiceTokenProvider.KeyVaultTokenCallback functionality. Updating my code to do something like this.

KeyVaultCredential creds = new KeyVaultCredential(new KeyVaultClient.AuthenticationCallback(async (authority, resource, scope) =>
{
    AzureServiceTokenProvider tokenProvider = new AzureServiceTokenProvider();
    string token = await tokenProvider.GetAccessTokenAsync(resource);
    return token;
}));

using(HttpClient client = new HttpClient())
using (KeyVaultClient keyVault = new KeyVaultClient(creds, client))
{
    SecretBundle sec = await keyVault.GetSecretAsync(secretUri);
}

instead of like this:

AzureServiceTokenProvider tokenProvider = new AzureServiceTokenProvider();

using (KeyVaultClient keyVault = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback((tokenProvider.KeyVaultTokenCallback))))
{
    SecretBundle sec = await keyVault.GetSecretAsync(secretUri);
}

seems to have resolved my issue.

0reactions

heathscommented, Feb 10, 2020

Closing as this issue seems to be resolved or worked around. The older Microsoft.Azure.KeyVault* packages will receive critical fixes, and customers are encouraged to update to the newer Azure.Security.KeyVault.* packages. See https://aka.ms/azsdkvalueprop for a growing list of reasons, including better HTTP pipelines that also resolve this issue.