[QUERY] Azure.Messaging.ServiceBus - How does an exception occurring in GetTokenAsync handler get handled by the library

Query/Question How can we help?

We use the TokenCredential overload when creating a ServiceBusClient to manage auth for use with service bus. We recently started to see some intermittent issues when calling _confidentialClient.AcquireTokenForClient to retrieve the access token needed for ServiceBus. Sometime this call starts to return 503’s which we have a policy in place to retry but occasionally it goes over the retry limit we have set in place and we let the exception bubble after that into Azure.Messaging.ServiceBus library.

Looking at Azure.Messaging.ServiceBus and how it handles this I’ve traced it down to https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/servicebus/Azure.Messaging.ServiceBus/src/Amqp/AmqpConnectionScope.cs#L915 throwing in this method. This method catches the exception and sets the refreshTimer’s dueTime to Timeout.Infinte. There is a comment present that indicates the operation would get retried but I’m having trouble following how this works in regards to an exception being thrown at this point.

What is the correct way to handle this scenario and does anything else need to be done to ensure proper retries. Should we the caller keep retrying on 503’s indefinitely and not let the exception bubble up at all? Or does things get handled properly already and nothing needs to be done?

Here is stack from when issue occured

MSAL.NetCore.4.19.0.0.MsalServiceException: 
	ErrorCode: service_not_available
Microsoft.Identity.Client.MsalServiceException: Service is unavailable to process the request
   at Microsoft.Identity.Client.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ICoreLogger logger, Boolean doNotThrow, Boolean retry)
   at Microsoft.Identity.Client.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ICoreLogger logger, Boolean doNotThrow, Boolean retry)
   at Microsoft.Identity.Client.Http.HttpManager.SendPostAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, ICoreLogger logger)
   at Microsoft.Identity.Client.Http.HttpManager.SendPostAsync(Uri endpoint, IDictionary`2 headers, IDictionary`2 bodyParameters, ICoreLogger logger)
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync[T](Uri endPoint, HttpMethod method, RequestContext requestContext, Boolean expectErrorsOn200OK, Boolean addCommonHeaders)
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.GetTokenAsync(Uri endPoint, RequestContext requestContext, Boolean addCommonHeaders)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, String scopeOverride, String tokenEndpointOverride, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.SendTokenRequestAsync(String tokenEndpoint, IDictionary`2 additionalBodyParameters, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
   at SkuVault.ServiceBus.Management.ServiceBusTokenService.<>c__DisplayClass11_0.<<AcquireTokenAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at SkuVault.ServiceBus.Management.ServiceBusTokenService.AcquireTokenAsync(CancellationToken token, List`1 scopes)
   at SkuVault.ServiceBus.Management.ServiceBusTokenService.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.CbsTokenProvider.GetTokenAsync(Uri namespaceAddress, String appliesTo, String[] requiredClaims)
   at Microsoft.Azure.Amqp.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult)
   at Microsoft.Azure.Amqp.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpCbsLink.<>c__DisplayClass4_0.<SendTokenAsync>b__1(IAsyncResult a)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.RequestAuthorizationUsingCbsAsync(AmqpConnection connection, CbsTokenProvider tokenProvider, Uri endpoint, String[] audience, String[] requiredClaims, TimeSpan timeout)
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.<>c__DisplayClass64_0.<<CreateAuthorizationRefreshHandler>b__0>d.MoveNext()
	StatusCode: 503 
	ResponseBody:  
	Headers: Cache-Control: private
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
client-request-id: 44390ff3-e528-4a60-9630-8338a2b00333
x-ms-request-id: e7f01568-0a41-4e9d-a5fa-2da1f017fd00
x-ms-ests-server: 2.1.11496.7 - NCUS ProdSlices
Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; httponly, stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Date: Tue, 23 Feb 2021 15:03:12 GMT

Environment:

• Azure.Messaging.ServiceBus 7.0.0
• Windows .net core 3.1

[chris-skuvault]

I see what you mean. As it is now we just recreate the receiver when we get this and that seems to work fine for us. It sounds like all of my concerns and questions have been addressed. Thanks so much!

Going to close this ticket.

[JoshLove-msft]

Yes, with sessions, we don’t reconnect the link. This is so that we can provide more deterministic behavior in the rest of the operations in a session receiver. Imagine that we did attempt to re-connect - you would need to handle the scenario where a SessionCannotBeLocked error is thrown, (which might be confusing given that you just had the session lock) in addition to a ServiceTimeout error (since there may be no more messages in that session) when doing ANY operation with the SessionReceiver. Rather than spread those exceptions out to all operations in the session receiver, we decided to have a central place where these are expected outcomes - when first accepting a session.