[ServiceBus] Queue receiver LinkDetach with UnknownError does not retry

• Package Name: azure-servicebus
• Package Version: 7.6.0
• Operating System: Linux (Docker python:3.9.10-slim)
• Python Version: 3.9.10

Describe the bug

23/02/2022 10:34:20 | INFO | 10 | uamqp.c_uamqp | work_async | Link ATTACH frame missing source and/or target. DETACH pending.
23/02/2022 10:34:20 | INFO | 10 | uamqp.c_uamqp | work_async | b'Cannot get initial delivery count' (b'/data/src/vendor/azure-uamqp-c/src/link.c':b'link_frame_received':343)
23/02/2022 10:34:20 | INFO | 10 | uamqp.receiver | _state_changed | Receiver link failed to open - expecting to receive DETACH frame.
23/02/2022 10:34:20 | INFO | 10 | uamqp.receiver | _detach_received | Received Link detach event: b'amqp:internal-error'
Link: b'receiver-link-8d043cc4-dbb7-4e18-ad6e-a4e9e4adcd65'
Description: b'The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:cf97e8a80d2c44178390b514fd6e98b1_G4, SystemTracker:gateway7, Timestamp:2022-02-23T22:34:20'
Details: None
Retryable: True
Connection: b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'
23/02/2022 10:34:20 | DEBUG | 10 | uamqp.c_uamqp | work_async | Deallocating cError
23/02/2022 10:34:20 | WARNING | 10 | uamqp.receiver | get_state | LinkDetach('ErrorCodes.InternalServerError: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:cf97e8a80d2c44178390b514fd6e98b1_G4, SystemTracker:gateway7, Timestamp:2022-02-23T22:34:20')
23/02/2022 10:34:20 | DEBUG | 10 | uamqp.c_uamqp | destroy | Destroying cMessageReceiver
23/02/2022 10:34:20 | DEBUG | 10 | uamqp.c_uamqp | destroy | Destroying cLink
23/02/2022 10:34:21 | INFO | 10 | uamqp.async_ops.client_async | close_async | CBS session pending b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'.
23/02/2022 10:34:21 | INFO | 10 | uamqp.async_ops.client_async | close_async | Closing exclusive connection b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'.
23/02/2022 10:34:21 | DEBUG | 10 | uamqp.async_ops.connection_async | destroy_async | Unlocked connection b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93' to close.
23/02/2022 10:34:21 | INFO | 10 | uamqp.async_ops.connection_async | _close_async | Shutting down connection b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'.
23/02/2022 10:34:21 | INFO | 10 | uamqp.authentication.cbs_auth_async | close_authenticator_async | Shutting down CBS session on connection: b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'.
23/02/2022 10:34:21 | DEBUG | 10 | uamqp.c_uamqp | close_authenticator_async | Destroying CBSTokenAuth for connection b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'
23/02/2022 10:34:21 | INFO | 10 | uamqp.authentication.cbs_auth_async | close_authenticator_async | Auth closed, destroying session on connection: b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'.
23/02/2022 10:34:21 | DEBUG | 10 | uamqp.c_uamqp | destroy_async | Destroying cSession
23/02/2022 10:34:21 | INFO | 10 | uamqp.authentication.cbs_auth_async | close_authenticator_async | Finished shutting down CBS session on connection: b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'.
23/02/2022 10:34:21 | DEBUG | 10 | uamqp.c_uamqp | _close_async | Destroying Connection
23/02/2022 10:34:21 | INFO | 10 | uamqp.connection | _state_changed | Connection b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93' state changed from <ConnectionState.OPENED: 9> to <ConnectionState.END: 13>
23/02/2022 10:34:21 | DEBUG | 10 | uamqp.c_uamqp | close | Destroying XIO
23/02/2022 10:34:21 | DEBUG | 10 | uamqp.c_uamqp | close | Destroying XIO
23/02/2022 10:34:21 | DEBUG | 10 | uamqp.c_uamqp | close | Destroying SASLMechanism
23/02/2022 10:34:21 | INFO | 10 | uamqp.async_ops.connection_async | _close_async | Connection shutdown complete b'SBReceiver-2c59d34f-76b1-4beb-8ac8-f1618e95fb93'.
23/02/2022 10:34:21 | DEBUG | 10 | uamqp | deinitialize | Deinitializing platform.
23/02/2022 10:34:21 | INFO | 10 | azure.servicebus.aio._base_handler_async | _handle_amqp_exception_with_condition | AMQP error occurred: (LinkDetach('ErrorCodes.InternalServerError: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:cf97e8a80d2c44178390b514fd6e98b1_G4, SystemTracker:gateway7, Timestamp:2022-02-23T22:34:20')), condition: (<ErrorCodes.InternalServerError: b'amqp:internal-error'>), description: (b'The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:cf97e8a80d2c44178390b514fd6e98b1_G4, SystemTracker:gateway7, Timestamp:2022-02-23T22:34:20').
Process IRCProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/azure/servicebus/aio/_base_handler_async.py", line 246, in _do_retryable_operation
    return await operation(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/azure/servicebus/aio/_servicebus_receiver_async.py", line 245, in _iter_next
    await self._open()
  File "/usr/local/lib/python3.9/site-packages/azure/servicebus/aio/_servicebus_receiver_async.py", line 350, in _open
    while not await self._handler.client_ready_async():
  File "/usr/local/lib/python3.9/site-packages/uamqp/async_ops/client_async.py", line 358, in client_ready_async
    if not await self._client_ready_async():
  File "/usr/local/lib/python3.9/site-packages/uamqp/async_ops/client_async.py", line 826, in _client_ready_async
    if self.message_handler.get_state() == constants.MessageReceiverState.Error:
  File "/usr/local/lib/python3.9/site-packages/uamqp/receiver.py", line 263, in get_state
    raise self._error
uamqp.errors.LinkDetach: ErrorCodes.InternalServerError: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:cf97e8a80d2c44178390b514fd6e98b1_G4, SystemTracker:gateway7, Timestamp:2022-02-23T22:34:20

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/src/bots/bot_manager.py", line 87, in run
    self.bot.start()
  File "/src/bots/irc_bot.py", line 55, in start
    self._loop.run_until_complete(asyncio.gather(servicebus_task, super_start_future, loop=self._loop))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/src/bots/irc_bot.py", line 60, in receive_servicebus_queue
    async for message in receiver:
  File "/usr/local/lib/python3.9/site-packages/azure/servicebus/aio/_servicebus_receiver_async.py", line 214, in __anext__
    message = await self.receiver._inner_anext()
  File "/usr/local/lib/python3.9/site-packages/azure/servicebus/aio/_servicebus_receiver_async.py", line 233, in _inner_anext
    return await self._do_retryable_operation(self._iter_next)
  File "/usr/local/lib/python3.9/site-packages/azure/servicebus/aio/_base_handler_async.py", line 250, in _do_retryable_operation
    last_exception = await self._handle_exception(exception)
  File "/usr/local/lib/python3.9/site-packages/azure/servicebus/aio/_base_handler_async.py", line 198, in _handle_exception
    raise error
azure.servicebus.exceptions.ServiceBusError: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:cf97e8a80d2c44178390b514fd6e98b1_G4, SystemTracker:gateway7, Timestamp:2022-02-23T22:34:20 Error condition: ErrorCodes.InternalServerError.

To Reproduce Use this code:

    async def receive_servicebus_queue(self):
        receiver = self.servicebus_client.get_queue_receiver(queue_name=self.listen_queue_name)
        async for message in receiver:
            logger.info(f'Received message from service bus: {str(message)}')

The script waits at the 3rd line: async for message in receiver:. Sometime later, it might produce this error. I don’t know how to reproduce this error as it happens in production.

Expected behavior The InternalServerError should be a retriable error.

[mbelak-dtml]

Hi @swathipil, any plans for releasing the fix for this issue? It’s blocking me since I don’t want implement a workaround for an issue which is already fixed. Thanks!

[efehana]

No problem it’s not blocking me since I do the retry operation for these kinds of errors.