[Azure.Identity] Suddenly stopped working and can't connect to Azure Key-Vault Secrets

Describe the bug We have deployed our .Net microservices on AKS where the code should go get some credentials from Key-Vault Secrets.

So we have deployed successfully it worked about a week and now we are facing this issue that when a service restarts (doesn’t need to be a new deployment, the same pod) it fails to connect to Key-Vault throwing the following error:

P.S. We didn’t change any config or update the AKS version.

Unhandled exception. Azure.Identity.CredentialUnavailableException: DefaultAzureCredential failed to retrieve a token from the included credentials.
- EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
- ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"}

Headers:
Server: IMDS/150.870.65.511
Date: Tue, 15 Jun 2021 12:57:17 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 168

- Operating system Linux 5.4.0-1046-azure #48~18.04.1-Ubuntu SMP Tue Apr 13 19:41:38 UTC 2021 isn't supported.
- Stored credentials not found. Need to authenticate user in VSCode Azure Account.
- Azure CLI not installed
- PowerShell is not installed.
 ---> System.AggregateException: Multiple exceptions were encountered while attempting to authenticate. (EnvironmentCredential authentication unavailable. Environment variables are not fully configured.) (ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"}

Headers:
Server: IMDS/150.870.65.511
Date: Tue, 15 Jun 2021 12:57:17 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 168
) (Operating system Linux 5.4.0-1046-azure #48~18.04.1-Ubuntu SMP Tue Apr 13 19:41:38 UTC 2021 isn't supported.) (Stored credentials not found. Need to authenticate user in VSCode Azure Account.) (Azure CLI not installed) (PowerShell is not installed.)
 ---> Azure.Identity.CredentialUnavailableException: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.EnvironmentCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.EnvironmentCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"}

Headers:
Server: IMDS/150.870.65.511
Date: Tue, 15 Jun 2021 12:57:17 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 168

   at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #2) Azure.Identity.CredentialUnavailableException: Operating system Linux 5.4.0-1046-azure #48~18.04.1-Ubuntu SMP Tue Apr 13 19:41:38 UTC 2021 isn't supported.
   at Azure.Identity.VisualStudioCredential.GetTokenProviderPath()
   at Azure.Identity.VisualStudioCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.VisualStudioCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.VisualStudioCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #3) Azure.Identity.CredentialUnavailableException: Stored credentials not found. Need to authenticate user in VSCode Azure Account.
 ---> System.DllNotFoundException: Unable to load shared library 'libsecret-1.so.0' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibsecret-1.so.0: cannot open shared object file: No such file or directory
   at Azure.Identity.LinuxNativeMethods.Imports.secret_schema_new(String name, Int32 flags, String attribute1, Int32 attribute1Type, String attribute2, Int32 attribute2Type, IntPtr end)
   at Azure.Identity.LinuxNativeMethods.secret_schema_new(String name, SecretSchemaFlags flags, String attribute1, SecretSchemaAttributeType attribute1Type, String attribute2, SecretSchemaAttributeType attribute2Type)
   at Azure.Identity.LinuxVisualStudioCodeAdapter.GetLibsecretSchema()
   at Azure.Identity.LinuxVisualStudioCodeAdapter.GetCredentials(String serviceName, String accountName)
   at Azure.Identity.VisualStudioCodeCredential.GetStoredCredentials(String environmentName)
   --- End of inner exception stack trace ---
   at Azure.Identity.VisualStudioCodeCredential.GetStoredCredentials(String environmentName)
   at Azure.Identity.VisualStudioCodeCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.VisualStudioCodeCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.VisualStudioCodeCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #4) Azure.Identity.CredentialUnavailableException: Azure CLI not installed
   at Azure.Identity.AzureCliCredential.RequestCliAccessTokenAsync(Boolean async, String[] scopes, CancellationToken cancellationToken)
   at Azure.Identity.AzureCliCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.AzureCliCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.AzureCliCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #5) Azure.Identity.CredentialUnavailableException: PowerShell is not installed.
   at Azure.Identity.AzurePowerShellCredential.RequestAzurePowerShellAccessTokenAsync(Boolean async, String[] scopes, CancellationToken cancellationToken)
   at Azure.Identity.AzurePowerShellCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.AzurePowerShellCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.AzurePowerShellCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

   --- End of inner exception stack trace ---
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthenticateRequestAsync(HttpMessage message, Boolean async, AuthenticationChallenge challenge)
   at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.ProcessCoreAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.GetPageAsync[T](Uri firstPageUri, String nextLink, Func`1 itemFactory, String operationName, CancellationToken cancellationToken)
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+MoveNext()
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at CodePlayground.Program.Main(String[] args) in /src/CodePlayground/Program.cs:line 34

C# Code to connect to Key-Vault Program.cs:

var keyValueSecretURL = _configuration["KeyVaultSecretURL"];
if (!string.IsNullOrWhiteSpace(keyValueSecretURL))
{
    var secretClient = new SecretClient(
        new Uri(keyValueSecretURL),
        new DefaultAzureCredential());
    config.AddAzureKeyVault(secretClient, new KeyVaultSecretManager());
}

appsettings.json:

"KeyVaultSecretURL": "The KV URL"

Dockerfile base images:

FROM mcr.microsoft.com/dotnet/aspnet:3.1-buster-slim AS base
...
FROM mcr.microsoft.com/dotnet/sdk:3.1-buster AS build

And From Azure portal, Key Vault, Settings, Access Policies I added the Kubernetes AgentPool With full access.

Environment:

• Name and version of the Library package used: [Azure.Identity 1.2.2, 1.2.3, 1.3.0, 1.4.0 ], [Azure.Extensions.AspNetCore.Configuration.Secrets 1.0.2, 1.2.1]
• Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): [e.g. Dockerized .Net Core 3.1]
• AKS Version: 1.19.7
• IDE and version : [e.g. Visual Studio 16.10.1]

[aaronsarkissian]

@schaabs @pakrym my issue still peersisted and it was not stable. But instead of giving the VMSS direct access to KV (because it was not scalable and with each node I had to add it in KV), I just added the following code part in the snippet below: new DefaultAzureCredentialOptions { ManagedIdentityClientId = _configuration["MANAGED_IDENTITY"] }

Full code:

var secretClient = new SecretClient(
  new Uri(keyValueSecretURL),
  new DefaultAzureCredential(new DefaultAzureCredentialOptions { ManagedIdentityClientId = _configuration["MANAGED_IDENTITY"] }));

That MANAGED_IDENTITY value is the Client ID of the Managed Identity from MC_…_AKS Resource Group.

P.S.: The MC_…_AK Resource Group is the default created RG of AKS.

Not sure if this is the best solution, but so far it worked stable.

[pakrym]

Getting the same error when using managed identity in AKS.

   ---> (Inner Exception #1) Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
      Status: 400 (Bad Request)

      Content:
      {"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"}

      Headers:
      Server: IMDS/150.870.65.514
      Date: Tue, 31 Aug 2021 00:03:03 GMT
      Content-Type: application/json; charset=utf-8
      Content-Length: 168

         at Azure.Identity.ImdsManagedIdentitySource.CreateRequest(String[] scopes)
         at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
         at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---