[QUERY] ServiceBus QuotaExceeded when creating Session Receiver

Library name and version

Azure.Messaging.ServiceBus 7.4.0

Query/Question

We seem to have started hitting a limit with a Quota on the number of handlers that are getting spun up for AMPQ connections as shown by the stack below

Azure.Messaging.ServiceBus.ServiceBusException: Cannot allocate more handles. The maximum number of handles is 4999. (QuotaExceeded)
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.CreateReceivingLinkAsync(String entityPath, String identifier, AmqpConnection connection, Uri endpoint, TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String sessionId, Boolean isSessionReceiver, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.OpenReceiverLinkAsync(String identifier, String entityPath, TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String sessionId, Boolean isSessionReceiver, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.OpenReceiverLinkAsync(TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, Boolean isSessionReceiver, String identifier)
   at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.<>c__DisplayClass68_0.<<OpenLinkAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation(Func`2 operation, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation(Func`2 operation, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.OpenLinkAsync(CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusReceiver.OpenLinkAsync(CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSessionReceiver.CreateSessionReceiverAsync(String entityPath, ServiceBusConnection connection, IList`1 plugins, ServiceBusSessionReceiverOptions options, String sessionId, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusClient.AcceptSessionAsync(String queueName, String sessionId, ServiceBusSessionReceiverOptions options, CancellationToken cancellationToken)

We use one ServiceBusClient for all of the session receivers we have running and we may have started running more than 5000 session receivers concurrently as when we decreased the amount of receivers we were running this error resolved. I’m trying to understand why this might occur because I was under the impression that one client reuses the same ampq connection for any amount of receivers.

I can’t find a whole lot of information on what exactly this error indicates but it seems like its happening on creating a AMPQ receiving link. Can someone help me understand what might be the issue here? Does this indicate that we are hitting the 5000 concurrent connections on a namespace? In telemetry it shows only 1 active connection to the namespace. I’m just trying to understand why this started happening so I can figure out steps to resolve.

Environment

.Net 5

[royramsta]

I took ownership of the support request, I will update here when Service Bus product team shares their response.

[chris-skuvault]

This has been answered by support so want to follow up with information that was learned. Each ServiceBusClient creates an AMPQ connection. The AMPQ connection has limitations on the number of links that can be opened. The limit seems to be 4999. Documentation is going to be updated to help provide more information about this as it doesn’t currently seem to be documented.

The receiver itself takes up 2 links, one for service bus sending a message to it and one for the receiver replying to service bus with a response to renew, complete, abandon, etc. Which is why when we got close to 2.5k receivers running we started to hit the quota of 4999 AMPQ links.

The solution that was recommended was to increase the number of AMPQ connections being used and distribute the load across more AMPQ connections. We decided to create a separate connection for each sender and a separate connection per 100 sessions that were running. Since we have 9 queues associated with each session then a single AMPQ connection will get 900 receivers running on it which should allow us plenty of room to not hit that AMPQ link quota. In testing when switching to this implementation we haven’t seen any link quota issues like before.

This documentation https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-protocol-guide#create-message-receiver was helpful to understand how the AMPQ links worked to understand the issue we were running into with the link quota.