[BUG] SNAT Port Exhaustion with Azure Service Bus

Describe the bug I’m using a Linux Web App that connects to an Azure Service Bus to send messages. The web app is causing SNAT port exhaustion connecting to servicebus.windows.net. It seems connections to servicebus.windows.net are not being reused or closed correctly. Messages are received in another app service. We send under a few hundred messages per hour.

Expected behavior Service Bus connections are reused or closed after use.

Actual behavior (include Exception or Stack Trace) The app is having issues with SNAT port exhaustion causing crashing.

To Reproduce In startup.cs, registration:

services.AddAzureClients(clientsBuilder =>
                {
                    clientsBuilder.AddServiceBusClient(Configuration.GetValue<string>("Queue:ServiceBusConnectionString"))
                      // (Optional) Provide name for instance to retrieve by with DI
                      //.WithName("Client1Name")
                      // (Optional) Override ServiceBusClientOptions (e.g. change retry settings)
                      .ConfigureOptions(options =>
                      {
                          //options.RetryOptions.Delay = TimeSpan.FromMilliseconds(50);
                          options.RetryOptions.MaxDelay = TimeSpan.FromSeconds(5);
                          options.RetryOptions.MaxRetries = 3;
                      });
                });

All queue message creation goes through this class:

    public class QueueService : IQueueService
    {
        private readonly IOptions<AppSettings> options;
        private readonly ILogger<QueueService> logger;
        private readonly ServiceBusClient serviceBusClient;

        public QueueService(IOptions<AppSettings> options, ILogger<QueueService> logger, ServiceBusClient serviceBusClient)
        {
            this.options = options;
            this.logger = logger;
            this.serviceBusClient = serviceBusClient;
        }

        public async Task Enqueue(string queue, string label, string message, bool throwErrors = false)
        {
            if ((options.Value?.Queue?.ServiceBusConnectionString).IsNullOrEmpty())
            {
                logger.LogError($"No service bus connection set for {label}: {message}");
                if (throwErrors)
                {
                    throw new Exception($"No service bus connection set for {label}: {message}");
                }
                return;
            }
            try
            {
                //https://docs.microsoft.com/en-us/dotnet/api/overview/azure/messaging.servicebus-readme-pre
                //Sender should not be disposed
                var sender = serviceBusClient.CreateSender(queue);
                
                var queueMessage = new ServiceBusMessage(Encoding.UTF8.GetBytes(message));
                queueMessage.Subject = label;
                await sender.SendMessageAsync(queueMessage);
            }
            catch (Exception e)
            {
                logger.LogError(e, $"Queue Error for {label}: {message}");
                if (throwErrors)
                {
                    throw;
                }
            }
        }

Environment: Assembly Azure.Messaging.ServiceBus, Version=7.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8 Azure Web App on Linux App Service .NET 5.0

[mjkent]

This seems to be working better. We’re deploying to test and will deploy to prod later today. Thank you again for your help!

services.AddSingleton<IQueueSenderCache, QueueSenderCache>();

    public class QueueService : IQueueService
    {
        private readonly IOptions<AppSettings> options;
        private readonly ILogger<QueueService> logger;
        private readonly IQueueSenderCache queueSenderCache;

        public QueueService(IOptions<AppSettings> options, ILogger<QueueService> logger, IQueueSenderCache queueSenderCache)
        {
            this.options = options;
            this.logger = logger;
            this.queueSenderCache = queueSenderCache;
        }

        public async Task Enqueue(string queue, string label, string message, bool throwErrors = false)
        {
            if ((options.Value?.Queue?.ServiceBusConnectionString).IsNullOrEmpty())
            {
                logger.LogError($"No service bus connection set for {label}: {message}");
                if (throwErrors)
                {
                    throw new Exception($"No service bus connection set for {label}: {message}");
                }
                return;
            }
            try
            {
                var sender = queueSenderCache.GetSender(queue);
                
                var queueMessage = new ServiceBusMessage(Encoding.UTF8.GetBytes(message));
                queueMessage.Subject = label;
                await sender.SendMessageAsync(queueMessage);
            }
            catch (Exception e)
            {
                logger.LogError(e, $"Queue Error for {label}: {message}");
                if (throwErrors)
                {
                    throw;
                }
            }
        }


    }

public class QueueSenderCache : IQueueSenderCache
    {
        private readonly ServiceBusClient serviceBusClient;
        public ConcurrentDictionary<string, ServiceBusSender> serviceBuses = new ConcurrentDictionary<string, ServiceBusSender>();

        //WARNING:SINGLETON
        public QueueSenderCache(ServiceBusClient serviceBusClient)
        {
            this.serviceBusClient = serviceBusClient;
        }

        public ServiceBusSender GetSender(string queueOrTopicName)
        {
            //https://docs.microsoft.com/en-us/dotnet/api/overview/azure/messaging.servicebus-readme-pre
            //Sender should not be disposed
            return serviceBuses.GetOrAdd(queueOrTopicName, (queueOrTopicName) =>  serviceBusClient.CreateSender(queueOrTopicName) );
        }
    }

[mjkent]

If someone else stumbles upon this, we found a few ways to fix this:

1. Switch to Windows App Service
2. Switch to NAT Gateway
3. Upgrade to Net 6.0

🤷