Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TaskCanceledException caused by ManagedIdentityCredential authentication failure when calling BlobContainerClient.CreateIfNotExists

See original GitHub issue

Query/Question

We tried to connect to storage container and queues with Managed Identity (system assigned) in our Webjob. Our Webjob tasks failed randomly but frequently for TaskCanceledException caused by ManagedIdentityCredential authentication failure. According the ApplicationInsights, it seems the exceptions were thrown when calling BlobContainerClient.CreateIfNotExists().

We assigned Storage Queue Data Contributor and Storage Blob Data Contributor roles to the AppService. We don’t think this is a permission issue since we didn’t see the failure constantly. The errors happened on different target storage containers in the same storage account. Retry a failed Webjob task may or may not get the same error. The same error may or may not happen during the next Webjob task with the same target storage account.

(More error details are attached below. )

Our current walk around is reverting to normal authentication (i.e. connect with connection string).

We cannot reproduce the same error in our test environment. We are wondering what’s the cause of this error and how to fix it.

Environment:

  • Packege version:
    • Azure.Storage.Blobs 12.9.1
    • Azure.Identity 1.4.0
  • Hosting platform: Azure AppService (ASP.NET V4.8)
  • IDE and version : Version 16.10.2

Screen Shot AuthError

Error for ManagedIdentityCredential.GetToken

Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. ---> System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. ---> System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at Azure.Core.CancellationHelper.ThrowIfCancellationRequested(CancellationToken cancellationToken)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RequestActivityPolicy.<ProcessAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
   --- End of inner exception stack trace ---
   at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully()
   at Azure.Core.Pipeline.LoggingPolicy.<ProcessAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
   --- End of inner exception stack trace ---
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Identity.ManagedIdentitySource.<AuthenticateAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Identity.ManagedIdentityClient.<AuthenticateAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Identity.ManagedIdentityCredential.<GetTokenImplAsync>d__9.MoveNext()
   --- End of inner exception stack trace ---
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.ManagedIdentityCredential.<GetTokenImplAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Identity.ManagedIdentityCredential.<GetTokenAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Identity.DefaultAzureCredential.<GetTokenFromSourcesAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext()

Full StackTrace from ApplicationInsights

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.27.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__15.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.27.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Inner exception System.AggregateException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at System.Threading.Tasks.Task.ThrowIfExceptional (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.Task.Wait (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at O365DC_WJ_CombineFiles.Functions.CombineFilesO365ForSpecificPrefix (O365DC-WJ-CombineFiles, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\panjerry\source\repos\DT-SIP-DPlat-SIP\CustomCollectors\O365DataCollection\O365DC-WJ-CombineFiles\Functions.cs:53)
   at lambda_method (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
   at Microsoft.Azure.WebJobs.Host.Executors.VoidMethodInvoker`2.InvokeAsync (Microsoft.Azure.WebJobs.Host, Version=3.0.27.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.27.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.27.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.27.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Inner exception Azure.Identity.AuthenticationFailedException handled at System.Threading.Tasks.Task.ThrowIfExceptional:
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Identity.ManagedIdentityCredential+<GetTokenImplAsync>d__9.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Identity.ManagedIdentityCredential+<GetTokenAsync>d__7.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Identity.DefaultAzureCredential+<GetTokenFromSourcesAsync>d__14.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask`1.get_Result (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Identity.DefaultAzureCredential+<GetTokenImplAsync>d__12.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Identity.DefaultAzureCredential+<GetTokenImplAsync>d__12.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Identity.DefaultAzureCredential+<GetTokenAsync>d__11.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+<GetHeaderValueFromCredentialAsync>d__12.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask`1.get_Result (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+<GetHeaderValueAsync>d__8.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+<GetHeaderValueAsync>d__8.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+<AuthenticateAndAuthorizeRequestAsync>d__12.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+<ProcessAsync>d__11.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy+<<ProcessAsync>g__ProcessAsyncInner|4_0>d.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Storage.Blobs.ContainerRestClient+<CreateAsync>d__6.MoveNext (Azure.Storage.Blobs, Version=12.9.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Storage.Blobs.BlobContainerClient+<CreateInternal>d__53.MoveNext (Azure.Storage.Blobs, Version=12.9.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Storage.Blobs.BlobContainerClient+<CreateIfNotExistsInternal>d__52.MoveNext (Azure.Storage.Blobs, Version=12.9.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Storage.Blobs.BlobContainerClient+<CreateIfNotExistsAsync>d__50.MoveNext (Azure.Storage.Blobs, Version=12.9.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at O365DC_WJ_CombineFiles.Functions+<CompressAndUploadToBlob>d__7.MoveNext (O365DC-WJ-CombineFiles, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\panjerry\source\repos\DT-SIP-DPlat-SIP\CustomCollectors\O365DataCollection\O365DC-WJ-CombineFiles\Functions.cs:226)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at O365DC_WJ_CombineFiles.Functions+<CombineJSONFilesBlob>d__5.MoveNext (O365DC-WJ-CombineFiles, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\panjerry\source\repos\DT-SIP-DPlat-SIP\CustomCollectors\O365DataCollection\O365DC-WJ-CombineFiles\Functions.cs:139)
Inner exception System.AggregateException handled at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow:
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Identity.ManagedIdentitySource+<AuthenticateAsync>d__9.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Identity.ManagedIdentityClient+<AuthenticateAsync>d__12.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Identity.ManagedIdentityCredential+<GetTokenImplAsync>d__9.MoveNext (Azure.Identity, Version=1.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception System.Threading.Tasks.TaskCanceledException handled at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext:
   at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.LoggingPolicy+<ProcessAsync>d__9.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception System.Threading.Tasks.TaskCanceledException handled at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout:
   at Azure.Core.CancellationHelper.ThrowIfCancellationRequested (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessInternal>d__7.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessAsync>d__6.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.RequestActivityPolicy+<ProcessAsync>d__9.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception System.Threading.Tasks.TaskCanceledException handled at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext:
   at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.LoggingPolicy+<ProcessAsync>d__9.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception System.Threading.Tasks.TaskCanceledException handled at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout:
   at Azure.Core.CancellationHelper.ThrowIfCancellationRequested (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessInternal>d__7.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessAsync>d__6.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.RequestActivityPolicy+<ProcessAsync>d__9.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception System.Threading.Tasks.TaskCanceledException handled at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext:
   at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.LoggingPolicy+<ProcessAsync>d__9.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully (System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception System.Threading.Tasks.TaskCanceledException handled at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout:
   at Azure.Core.CancellationHelper.ThrowIfCancellationRequested (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessInternal>d__7.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.HttpWebRequestTransport+<ProcessAsync>d__6.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.RequestActivityPolicy+<ProcessAsync>d__9.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext (Azure.Core, Version=1.16.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception Microsoft.ApplicationInsights.DataContracts.InnerExceptionCountExceededException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:13 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
christothescommented, Aug 23, 2021

As discussed offline, this sounds like a service issue that you’ll need to raise with them.

Can we have some insights to the offline discussion that took place? We are currently encountering the same issue.

The gist of it is that this is something happening with the managed identity endpoint and it would need to be diagnosed from that perspective.

0reactions
bradmouldercommented, Sep 8, 2021

@sasoripathos The fix for us was to upgrade our “Azure.Storage.Blobs” Version=“12.10.0-beta.2” we were hitting a known bug in the 12.9.1 version that was throwing an error for us.

We also saw some issues around our local Token Service (or thats what i assume that Get /msi/token call is) We were getting 500 errors on that call. only known fix is to delete and re-deploy the function app for us.

Read more comments on GitHub >

github_iconTop Results From Across the Web

DefaultAzureCredential fails when multiple accounts are ...
Environment variables are not fully configured. ManagedIdentityCredential authentication unavailable, no managed identity endpoint found.
Read more >
[BUG] DefaultAzureCredential fails when using Managed ...
Describe the bug I am trying to write a Blob from a function using a Managed Identity, through the DefaultAzureCredential.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[FEATURE REQ] Make it possible to mock SchemaRegistryClient

Next page

[BUG] Context propagation for eventhub between Java and .NET is broken