ServiceBus RenewLockAsync occurs too late

Library name and version

Azure.Messaging.ServiceBus 7.5.1

Describe the bug

We use the library Azure.Messaging.serviceBus inside a webjob project (.net core 3.1) to process long running operation raised by event message. We use the ServiceBusTrigger attribute on function and let the library renew the locktoken when needed. The service bus queue has a LongDuration set to 5 minutes and the ServiceBusOptions.MaxConcurrentCalls is set to 5 and MaxAutoLockRenewalDuration is set to 1hour.

Sometimes we observe than the RenewLockAsync occurs too late, like the a2ff7048-780d-400c-b5e3-e2bd1eb5006b behind

[2022-02-25 02:00:39] info: Azure.Messaging.ServiceBus[8]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: ReceiveBatchAsync done. Received '1' messages. LockTokens = <LockToken>a2ff7048-780d-400c-b5e3-e2bd1eb5006b</LockToken>
[2022-02-25 02:00:39] info: Azure.Messaging.ServiceBus[56]
      prmimportpivot-b6100f05-4c64-4c5b-a987-c155ef3ebc4e: Processor RenewMessageLock start. MessageCount = 1, LockToken = a2ff7048-780d-400c-b5e3-e2bd1eb5006b
[2022-02-25 02:00:39] info: Azure.Messaging.ServiceBus[102]
      prmimportpivot-b6100f05-4c64-4c5b-a987-c155ef3ebc4e: User message handler start: Message: SequenceNumber: 30117822508040715, LockToken: a2ff7048-780d-400c-b5e3-e2bd1eb5006b

...

[2022-02-25 02:05:29] info: Azure.Messaging.ServiceBus[31]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync start. MessageCount = 1, LockToken = a2ff7048-780d-400c-b5e3-e2bd1eb5006b

...

[2022-02-25 02:05:29] info: Azure.Messaging.ServiceBus[56]
      prmimportpivot-b6100f05-4c64-4c5b-a987-c155ef3ebc4e: Processor RenewMessageLock start. MessageCount = 1, LockToken = a2ff7048-780d-400c-b5e3-e2bd1eb5006b
[2022-02-25 02:05:29] info: Azure.Messaging.ServiceBus[31]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync start. MessageCount = 1, LockToken = a90e6e67-29b1-4be8-9c15-f1fd7b450071
[2022-02-25 02:05:29] info: Azure.Messaging.ServiceBus[32]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync done.
[2022-02-25 02:05:29] info: Azure.Messaging.ServiceBus[57]
      prmimportpivot-b6100f05-4c64-4c5b-a987-c155ef3ebc4e: Processor RenewMessageLock complete.

...

[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[31]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync start. MessageCount = 1, LockToken = a2ff7048-780d-400c-b5e3-e2bd1eb5006b
[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[32]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync done.
[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[57]
      prmimportpivot-b6100f05-4c64-4c5b-a987-c155ef3ebc4e: Processor RenewMessageLock complete.
[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[56]
      prmimportpivot-b6100f05-4c64-4c5b-a987-c155ef3ebc4e: Processor RenewMessageLock start. MessageCount = 1, LockToken = a2ff7048-780d-400c-b5e3-e2bd1eb5006b
[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[31]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync start. MessageCount = 1, LockToken = 35b54365-fd59-40bf-9ab0-0d985ebb005c
[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[31]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync start. MessageCount = 1, LockToken = 33f76443-0d9f-4dfb-a4f2-3151d0570049
[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[32]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync done.
[2022-02-25 02:10:19] info: Azure.Messaging.ServiceBus[57]
      prmimportpivot-b6100f05-4c64-4c5b-a987-c155ef3ebc4e: Processor RenewMessageLock complete.

...

[2022-02-25 02:15:20] info: Azure.Messaging.ServiceBus[31]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync start. MessageCount = 1, LockToken = a2ff7048-780d-400c-b5e3-e2bd1eb5006b
[2022-02-25 02:15:20] fail: Azure.Messaging.ServiceBus[33]
      prmimportpivot-e9cadf78-2862-4fb5-a40a-9c100f86fa48: RenewLockAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see https://aka.ms/ServiceBusExceptions . Reference:61393e9f-c8dd-41b4-8c83-59fe1c794bec, TrackingId:a6efd5c8-bb65-4958-94a2-f301b8c2f9cf_B2, SystemTracker:sbopera001:queue:prmimportpivot~127, Timestamp:2022-02-25T02:15:20 (MessageLockLost)
         at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.RenewMessageLockInternalAsync(Guid lockToken, TimeSpan timeout)
         at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.<>c.<<RenewMessageLockAsync>b__57_0>d.MoveNext()
      --- End of stack trace from previous location where exception was thrown ---
         at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
         at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
         at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.RenewMessageLockAsync(Guid lockToken, CancellationToken cancellationToken)
         at Azure.Messaging.ServiceBus.ServiceBusReceiver.RenewMessageLockAsync(Guid lockToken, CancellationToken cancellationToken).

The two first RenewLock occurs well 10seconds before the end of the LockDuration but the third occurs 1second after the end of LockDuration (too late). Apparently they are no way to fire the RenewLock with a parameter (20 seconds before for sample).

Our assumption is that when they are a lot of message to proceed the RenewLock task is delayed by others task.

Will be nice to be able to have option in ServiceBusOption to modify the Constants.MaximumRenewBufferDuration.Ticks (10 seconds)

Expected behavior

The RenewLockAsync must be fired before end of Service Bus queue LockDuration

Actual behavior

The RenewLockAsync occurs after the end of Service Bus Queue LockDuration

Reproduction Steps

Environment

Platform : Debian GNU/Linux 10 (buster) Hosting docker image : mcr.microsoft.com/dotnet/aspnet:3.1

[JoshLove-msft]

@hankbeasley, yes if the clock that the application is running on is not accurate (specifically if it is slow), then you may see the lock renewal not occurring in time. LockedUntil is set by azure servers.

[LZorglub]

Replacing Task.WaitAll(tasks, ctts) by await Task.WhenAll(tasks) in our main function do the job. After 6 days no RenewLock exception. Thanks for the hint.