ServiceBus: Async permanent receiver failing due to authorization timeout

• Package Name: azure-servicebus
• Package Version: 7.0.0
• Operating System: Linux Debian/Ubuntu
• Python Version: 3.7.9
• uamqp Version: 1.2.12

Describe the bug An async permanent service bus receiver is failing due to authorization timeout after some hours of operation.

The below debug logs should indicate that the token refreshing is working:

2020-12-24 06:35:36,199 uamqp.authentication.cbs_auth_async [INFO] Token on connection b'SBReceiver-REDACTED' will expire soon - attempting to refresh.
2020-12-24 06:35:36,199 azure.identity.aio._internal.decorators [INFO] ManagedIdentityCredential.get_token succeeded

But it is not. After a large amount of token refreshing attempts (at ~50ms intervals), the following exception is thrown, and the connection is closed:

2020-12-24 06:45:19,000 azure.servicebus.aio._base_handler_async [ERROR] Unexpected error occurred (TimeoutError('Authorization timeout.')). Handler shutting down.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/azure/servicebus/aio/_base_handler_async.py", line 181, in _do_retryable_operation
    return await operation(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/servicebus/aio/_servicebus_receiver_async.py", line 343, in _receive
    receiving = await amqp_receive_client.do_work_async()
  File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 378, in do_work_async
    if not await self.client_ready_async():
  File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 360, in client_ready_async
    if not await self.auth_complete_async():
  File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 346, in auth_complete_async
    raise TimeoutException("Authorization timeout.")
TimeoutError: Authorization timeout.

To Reproduce Steps to reproduce the behavior:

1. Create an async receiver using DefaultAzureCredential
2. Start an indefinite iteration to create a permanent receiver waiting for messages
3. After some hours of operation, the program will fail due to authorization timeout

Expected behavior Authorization refreshing is managed successfully by the library.

Code Basic code to reproduce the issue:

from datetime import datetime
from asyncio import get_event_loop, sleep
from azure.identity.aio import DefaultAzureCredential
from azure.servicebus.aio import ServiceBusClient
from azure.servicebus import ServiceBusReceiveMode

import logging

logging.basicConfig(level=logging.DEBUG, format="%(asctime)-15s %(name)s [%(levelname)s] %(message)s")

SB_NAMESPACE_FQDN = "..."
QUEUE_NAME = "..."


async def receive_forever():
    attempt = 0
    while True:
        start = datetime.now()
        try:
            attempt += 1
            logging.info(f"== Starting attempt={attempt}")

            async with DefaultAzureCredential(exclude_environment_credential=True) as credential:
                service_bus_client = ServiceBusClient(SB_NAMESPACE_FQDN, credential, logging_enable=True)
                async with service_bus_client:
                    queue_receiver = service_bus_client.get_queue_receiver(
                        QUEUE_NAME,
                        receive_mode=ServiceBusReceiveMode.RECEIVE_AND_DELETE,
                    )
                    async with queue_receiver:
                        while True:
                            messages = await queue_receiver.receive_messages(max_message_count=1, max_wait_time=30)
                            if messages:
                                logging.debug(f"Received: {messages[0]}")
        except Exception:
            logging.exception(f"== Running for {datetime.now()-start}")
            await sleep(10)


if __name__ == "__main__":
    loop = get_event_loop()
    loop.run_until_complete(receive_forever())

Please find attached an reduced output log.

reduced.log

[yunhaoling]

@danidelvalle , I’m currently running your code on a linux vm, let’s see if I could reproduce the issue.

btw, does this issue happen when you’re using a connection string?

[msftbot[bot]]

Hi, we’re sending this friendly reminder because we haven’t heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don’t hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!