[BUG] Many abandoned messages in queues/topics

Description Recently we migrated from WindowsAzure.ServiceBus to Azure.Messaging.ServiceBus. After deployment to production we noticed that processing of messages is slow down and there are many abandoned messages in queue/topics.

There is screenshot of metrics before deployment (WindowsAzure.ServiceBus):

And there is screenshot of current state (Azure.Messaging.ServiceBus):

We investigated the issue and found:

1. We do not abandon messages in code, it happens inside of Azure.Messaging.ServiceBus
2. Topics and queues are affected
3. We have the issue only if ServiceBusReceiverOptions.PrefetchCount is not defined.

Expected behavior No abandoned messages if we do not abandon them directly in code.

Actual behavior (include Exception or Stack Trace) Lot’s of abandoned massages came from the library.

To Reproduce I was able to reproduce the problem in console application:

	class Program
	{
		static void Main(string[] args)
		{
			var receiver = new Receiver4();
			receiver.ReadMessages().GetAwaiter().GetResult();			
		}
	}

    class Receiver4
    {
        private const string ConnectionString = "connection string";

        public async Task ReadMessages()
        {
            var sbOptions = new ServiceBusClientOptions()
            {
                RetryOptions = new ServiceBusRetryOptions
                {
                    Mode = ServiceBusRetryMode.Fixed,
                    Delay = TimeSpan.FromSeconds(1),
                    MaxDelay = TimeSpan.FromSeconds(5),
                    MaxRetries = 1,
                    TryTimeout = TimeSpan.FromSeconds(30)
                }
            };

            ServiceBusClient client = new ServiceBusClient(ConnectionString, sbOptions);

            ServiceBusReceiverOptions options = new ServiceBusReceiverOptions
            {
            };

            var receiver = client.CreateReceiver("alex-test", options);

            var receivedMessages = new List<ServiceBusReceivedMessage>();
            while (true)
            {
                Console.WriteLine("Reading");

                var messages = await receiver.ReceiveMessagesAsync(128, TimeSpan.FromSeconds(5)).ConfigureAwait(false);

                Console.WriteLine($"{messages.Count} were received");

                foreach (var message in messages)
                {
                    Console.WriteLine(message.Body.ToString());

                    receivedMessages.Add(message);
                }

                await Task.Delay(100);

                Console.WriteLine("Completing");

                foreach (var message in receivedMessages)
                {
                    try
                    {
                        await receiver.CompleteMessageAsync(message).ConfigureAwait(false);
                    }
                    catch (ServiceBusException ex) when (ex.Reason == ServiceBusFailureReason.MessageLockLost)
                    {
                        Console.WriteLine("MessageLockLost, {0}", message.Body.ToString());
                    }
                }

                receivedMessages.Clear();

                Console.WriteLine("Completed");

                ConsoleKeyInfo key = Console.ReadKey();

                if (key.KeyChar == 's')
                {
                    break;
                }
            }
        }
    }

Environment:

• Name and version of the Library package used: Azure.Messaging.ServiceBus 7.3.0
• Hosting platform or OS and .NET runtime version Azure Cloud Services
• IDE and version : Visual Studio 16.11.4

[JoshLove-msft]

Thanks, I was able to repro and am seeing the same behavior. I have reached out to the service team to find out more about this metric.

[Alex-Boo]

Unlock is a variation of abandon, and our metrics are reporting that operation as abandon. We will fix it in the service. It is not a client issue.

@yvgopal, thank you for explanation. Can we track somehow fixing this issue on server side?