[BUG] Disabled ServiceBus queue causes fixed retry after 300ms

Library name and version

Azure.Messaging.ServiceBus 7.6.0

Describe the bug

Disabled ServiceBus queue causes the ServiceBus client to retry consuming messages after a fixed retry inverval of ~300ms. If MaxConcurrentCalls is set in a ServiceBusProcessorOptions, it will try connect (and fail) that many times in parallel. This happens regardless of what the retry policy is configured to. In my case this results in a virtually infinite loop and causes the application to hog CPU usage if the queue is disabled.

Expected behavior

I expect the client to adhere to the retry policy and retry according to what the retry policy says if the queue is disabled.

Actual behavior

The client will retry consuming messages after ~300ms if the ServiceBus queue is disabled, regardless of what the retry policy says.

Reproduction Steps

Create a queue and disable it in in Azure ServiceBus, then try consume the messages using this .NET 6 app:

using Azure.Messaging.ServiceBus;

var clientOptions = new ServiceBusClientOptions
{
    RetryOptions = new ServiceBusRetryOptions
    {
        Delay = TimeSpan.FromSeconds(10),
        Mode = ServiceBusRetryMode.Exponential
    }
};
var client = new ServiceBusClient("Endpoint=sb://sb-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=secret", clientOptions);
var processorOptions = new ServiceBusProcessorOptions
{
    MaxConcurrentCalls = 100
};

var processor = client.CreateProcessor("some.queue.name", processorOptions);

processor.ProcessMessageAsync += (_ => Task.CompletedTask);
processor.ProcessErrorAsync += args =>
{
    Console.WriteLine($"ServiceBus adapter error: " + args.Exception.Message);
    return Task.CompletedTask;
};

await processor.StartProcessingAsync();

Console.ReadLine();

The application will spam the following message without a 10 second wait between each try:

ServiceBus adapter error: Messaging entity 'sb://sb-namespace.servicebus.windows.net/some.queue.name' is currently disabled. For more information please see https://aka.ms/ServiceBusExceptions . TrackingId:e5f67fd40bf64f088abb211a8a62ed5f_G12, SystemTracker:gateway7, Timestamp:2022-03-02T13:44:02 (MessagingEntityDisabled)

Environment

.NET SDK (reflecting any global.json):
 Version:   6.0.101
 Commit:    ef49f6213a

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.101\

Host (useful for support):
  Version: 6.0.1
  Commit:  3a25a7f1cc

[jsquire]

In this case, as Josh mentioned, MessagingEntityDisabled is not a retriable exception - which I had initially overlooked. When the service operation encounters the error, it does not implicitly retry and fails. This failure is observed by one of the processor tasks which intercepts the exception and surfaces it to your error handler.

The processor is built to be highly resilient in the face of errors. After notifying the application of the failure by invoking the error handler, the task will continue to try and make forward progress. The processor does not understand enough about the host environment or application to automatically apply a heuristic across error patterns to alter that behavior. Currently, it falls on the application to observe errors in the handler and make determinations about the overall state and whether the processor would not be able to recover.

In this scenario, because the error handler is awaited, your application could choose to introduce a delay, whether globally or per-handler, which would be respected and stop. It may also interpret the pattern of exceptions as fatal for the application and call StopProcessingAsync.

In the simple, per-handler case, this would look something like:

processor.ProcessErrorAsync += async args =>
{
    if (args.Exception is ServiceBusException { Reason: ServiceBusFailureReason.MessagingEntityDisabled })
    {
        await Task.Delay(TimeSpan.FromSeconds(30).ConfigureAwait(false);
    }

    Console.WriteLine($"ServiceBus adapter error: " + args.Exception.Message);
};

[martinmine]

Thanks a lot for the help! I will close this issue as I can deal with this problem in the ProcessErrorAsync as demoed above.