Intermittent Exceptions when calling DefaultAzureCredential

Describe the bug Since moving to the latest Azure Identity libraries we have been plagued with intermittent Exceptions when calling DefaultAzureCredential.GetTokenAsync. The code works fine 99% of the time, but sometimes we get one of the following two Exceptions. The code also then hangs soon afterwards - we haven’t any hard evidence to implicate Azure Identity in this but we have suspicions.

Expected behavior The code should reliably obtain a credential token from Azure. The code should be thread-safe.

Actual behavior (include Exception or Stack Trace) The call usually works, but occasionally it throws an Exception with one of the following stack traces. Either Collection was modified:

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
 at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
 at System.Collections.Generic.Dictionary`2.Enumerator.MoveNext()
 at Azure.Identity.CredentialPipeline.ScopeGroupHandler.Fail(String name, DiagnosticScope& scope, Exception exception)
 at Azure.Identity.CredentialDiagnosticScope.RegisterFailed(Exception ex)
 at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
 at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<AuthenticateRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<ProcessCoreAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipeline.<SendRequestAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__27.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__21`1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.Secrets.SecretClient.<GetSecretAsync>d__8.MoveNext()

Or NullReferenceException:

Azure.Identity.AuthenticationFailedException: DefaultAzureCredential authentication failed: Object reference not set to an instance of an object. ---> System.NullReferenceException: Object reference not set to an instance of an object.
 at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
 at Azure.Identity.CredentialPipeline.ScopeGroupHandler.CreateScope(String name)
 at Azure.Identity.CredentialPipeline.StartGetTokenScope(String fullyQualifiedMethod, TokenRequestContext context)
 at Azure.Identity.EnvironmentCredential.<GetTokenImplAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.EnvironmentCredential.<GetTokenAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenFromSourcesAsync>d__14.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of inner exception stack trace ---
 at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
 at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<AuthenticateRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<ProcessCoreAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Core.Pipeline.HttpPipeline.<SendRequestAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__27.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__21`1.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Azure.Security.KeyVault.Secrets.SecretClient.<GetSecretAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at

To Reproduce This is the code used to call GetSecretAsync, nothing fancy. It is called from a multi-threaded environment although the result is cached elsewhere to avoid quota / throttling issues on the KeyVault itself.

public async Task<string> GetConnectionStringAsync(CancellationToken cancellationToken = default)
{
  var client = new SecretClient(new Uri(mVaultUri), new DefaultAzureCredential());
  {
    var result = await client.GetSecretAsync(mSecretName, cancellationToken: cancellationToken).ConfigureAwait(false);
    return result.Value.Value;
  }
}

Environment:

• Name and version of the Library package used: [Azure.Identity 1.2.2, Azure.Security.KeyVault.Secrets 4.1.0]
• Hosting platform or OS and .NET runtime version (Windows Server 2016 and .NET Framework 4.7.2]

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[thecodetinker]

We’ve done some runs with the new code and so far, so good. Thanks for turning this around so quickly, much appreciated!