ServiceBusTrigger continously logs warnings about "The operation did not complete within the allocated time 00:01:00 for object receiverNNN"

We have a Function App running in Azure, with a ServiceBusTrigger configured. And while it is running, twice every minute, it will log a warning for:

RunOperation encountered an exception and will retry. Exception: Azure.Messaging.ServiceBus.ServiceBusException: The operation did not complete within the allocated time 00:01:00 for object receiver596. (ServiceTimeout)
 ---> System.TimeoutException: The operation did not complete within the allocated time 00:01:00 for object receiver596.
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.<>c.<OpenAsync>b__51_1(IAsyncResult r)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.OpenAmqpObjectAsync(AmqpObject target, TimeSpan timeout, CancellationToken cancellationToken, String entityPath)
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.OpenReceiverLinkAsync(String identifier, String entityPath, TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String sessionId, Boolean isSessionReceiver, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.OpenReceiverLinkAsync(TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String identifier, CancellationToken cancellationToken)
   at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.<>c.<<OpenLinkAsync>b__72_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

Category: Azure.Messaging.ServiceBus LogLevel: Warning EventName: RunOperationExceptionEncountered

I spent about 1 working-day troubleshooting this, until I realized it is actually fully functional, and I guess the behavior is part of the intented implementation. BUT logging this with loglevel warning should not be done then.

Nuget packages: Microsoft.Azure.WebJobs.Extensions.ServiceBus, 5.2.0 Microsoft.NET.Sdk.Functions, 4.0.1

[JoshLove-msft]

@JoshLove-msft I totally agree that these messages should be at the Debug level. I don’t want to have to filter out this namespace specifically at the Informational level as there may be other messages that I do want to log. As no action can be performed by the developer to change whether or not these messages would appear, I see them as diagnostic and part of normal correct operation rather than something which should be logged with such frequency.

I would agree that for this specific retriable error, logging as verbose would be sufficient. In general, I think retries for exceptions that are not expected during the normal application flow should be Informational. I will take a look and see what we can do to improve this.

[danielabbatt]

@JoshLove-msft I totally agree that these messages should be at the Debug level. I don’t want to have to filter out this namespace specifically at the Informational level as there may be other messages that I do want to log. As no action can be performed by the developer to change whether or not these messages would appear, I see them as diagnostic and part of normal correct operation rather than something which should be logged with such frequency.