[BUG] Message lock renewal not working for 10+ minutes processin durationg when using different `ServiceBusReceiver` instances for `CompleteMessageAsync` and `RenewMessageLockAsync`

Library name and version

Azure.Messaging.ServiceBus 7.8.0, 7.8.1 and 7.9.0

Describe the bug

When using transactions and invoking RenewMessageLockAsync on a different ServiceBusReceiver and processing durations are over 10 minutes then invoking CompleteMessageAsync CAN results in a MessageLockLost if that is the only message concurrently being processed.

A similar bug was reported earlier. This has been fixed in 7.8.0 when invoking invoking CompleteMessageAsync on the same receiver as CompleteMessageAsync.

• https://github.com/Azure/azure-sdk-for-net/issues/28025

Expected behavior

The client not gets into a faulty state where lock renewal seems to succeed but when CompleteMessageAsync is invoked that it fails with MessageLockLost.

Actual behavior

The client can get into a faulty state where:

• any message that will be ack’d via CompleteMessageAsync where processing is longer then the queue lock duration will fail
• any message that result in a lock renewal via RenewMessageLockAsync will incorrectly succeed

Reproduction Steps

var lockDuration = TimeSpan.FromMinutes(5);
var renewalInterval = lockDuration - TimeSpan.FromSeconds(10);
var queueName = "test-" + DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
var cs = Environment.GetEnvironmentVariable("AzureServiceBus_ConnectionString");
var admin = new ServiceBusAdministrationClient(cs);
await admin.CreateQueueAsync(new CreateQueueOptions(queueName) { LockDuration = lockDuration, AutoDeleteOnIdle = TimeSpan.FromMinutes(10) });
var sw = Stopwatch.StartNew();
using var logger = new AzureEventSourceListener((ea, m) => Console.Out.WriteLineAsync($"{sw.Elapsed,-15:g} [{ea.Level,-13}] {m}"), EventLevel.Verbose);
await using var client = new ServiceBusClient(cs, new ServiceBusClientOptions { EnableCrossEntityTransactions = true });
await using var sender = client.CreateSender(queueName);
await sender.SendMessageAsync(new ServiceBusMessage());
await using var receiver = client.CreateReceiver(queueName);

var message = await receiver.ReceiveMessageAsync(); Trace.Assert(message != null);
using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(11));
await RenewLockAsync(message!, cts.Token);
using var ts = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
await sender.SendMessageAsync(new ServiceBusMessage());
await receiver.CompleteMessageAsync(message);
ts.Complete();

async Task RenewLockAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken)
{
    var renewLockReceiver = client.CreateReceiver(queueName);

    while (!cancellationToken.IsCancellationRequested)
    {
        var delayTask = await Task.Delay(renewalInterval, cancellationToken)
            .ContinueWith(t => t, TaskContinuationOptions.ExecuteSynchronously);
        if (delayTask.IsCanceled) return;
        await renewLockReceiver.RenewMessageLockAsync(message);
    }
}

Environment

• net6
• windows 10

[ramonsmits]

Another way is to use it to receive other messages. In your case, you are letting the receiver go idle after receiving the initial message, and it gets closed after 10 minutes.

I think in our case that is actually happening. For example, allowing for a processing concurrency of 10 but then having only a single message in the queue. That means 9 receive operations are waiting for a message. If no messages are in the queue (meaning, the queue is more or less idle) this can happen too.

[ramonsmits]

@JoshLove-msft It seems my repro code was incorrect and I found the difference. I’ve updated the description with the following:

When using transactions and invoking RenewMessageLockAsync on a different ServiceBusReceiver and processing durations are over 10 minutes then invoking CompleteMessageAsync CAN results in a MessageLockLost if that is the only message concurrently being processed.

The important question is. Is this actually a supported scenario?

• If yes, then it should be fixed
• If no, then invoking RenewMessageLockAsync on a different ServiceBusReceiver instance should result in an exception.