[Service Bus]: DetachForcedError

• Package Name: @azure/service-bus
• Package Version: 7.5.1
• nodejs
  • version: 16

Describe the bug Sudden DetachForcedError happen across multiple servers (10 pods deployed on Kubernetes) listening on the same subscription. They then stop listening to the topics unless we restart the servers.

I have the extract of logs for one server with some DEBUG here :

DetachForcedError: The link 'G9:13642683:prd.gps-train.gps-tracker.messages/Subscriptions/processing-0f774707-92ad-a643-95b8-6e15eb539d7f' is force detached. Code: consumer(link35055). Details: InnerMessageReceiver was closed. TrackingId:638ebc4e000206d9000088ef62fbac03_G9_B32, SystemTracker:pzsbeotpr01:Topic:prd.gps-train.gps-tracker.messages|processing, Timestamp:2022-08-16T15:53:02
Exiting application...
2022-08-16T15:53:02.728Z rhea:io [connection-1] read 433 bytes
2022-08-16T15:53:02.728Z rhea:io [connection-1] got frame of size 433
2022-08-16T15:53:02.728Z rhea:frames [connection-1]:1 <- detach#16 {"closed":true,"error":{"condition":"amqp:link:detach-forced","description":"The link 'G9:13642683:prd.gps-train.gps-tracker.messages/Subscriptions/processing-0f774707-92ad-a643-95b8-6e15eb539d7f' is force detached. Code: consumer(link35055). Details: InnerMessageReceiver was closed. TrackingId:638ebc4e000206d9000088ef62fbac03_G9_B32, SystemTracker:pzsbeotpr01:Topic:prd.gps-train.gps-tracker.messages|processing, Timestamp:2022-08-16T15:53:02"}}
2022-08-16T15:53:02.734Z rhea:events [connection-1] Link got event: receiver_error
2022-08-16T15:53:02.743Z rhea-promise:receiver [connection-1] receiver 'prd.gps-train.gps-tracker.messages/Subscriptions/processing-0f774707-92ad-a643-95b8-6e15eb539d7f' got event: 'receiver_error'. Re-emitting the translated context.
2022-08-16T15:53:02.743Z rhea-promise:translate [connection-1] Translating the context for event: 'receiver_error'.
azure:service-bus:receiver:warning [connection-1|streaming:prd.gps-train.gps-tracker.messages/Subscriptions/processing-0f774707-92ad-a643-95b8-6e15eb539d7f] 'receiver_error' event occurred. The associated error is : ServiceBusError: DetachForcedError: The link 'G9:13642683:prd.gps-train.gps-tracker.messages/Subscriptions/processing-0f774707-92ad-a643-95b8-6e15eb539d7f' is force detached. Code: consumer(link35055). Details: InnerMessageReceiver was closed. TrackingId:638ebc4e000206d9000088ef62fbac03_G9_B32, SystemTracker:pzsbeotpr01:Topic:prd.gps-train.gps-tracker.messages|processing, Timestamp:2022-08-16T15:53:02
at translateServiceBusError (/node_modules/@groupeiot-itnovem/esncf-iot-servicebus/node_modules/@azure/service-bus/dist/index.js:4575:16)
at Receiver.StreamingReceiver._onAmqpError (/node_modules/@groupeiot-itnovem/esncf-iot-servicebus/node_modules/@azure/service-bus/dist/index.js:7082:33)
at Receiver.emit (node:events:527:28)
at emit (/node_modules/rhea-promise/dist/lib/util/utils.js:131:24)
at Object.emitEvent (/node_modules/rhea-promise/dist/lib/util/utils.js:142:9)
at Receiver.<anonymous> (/node_modules/rhea-promise/dist/lib/link.js:284:25)
at Receiver.emit (node:events:527:28)
at Receiver.link.dispatch (/node_modules/rhea/lib/link.js:59:37)
at Receiver.<anonymous> (/node_modules/dd-trace/packages/datadog-plugin-rhea/src/index.js:91:23)
at Receiver.link.on_detach (/node_modules/rhea/lib/link.js:162:32) {
retryable: true,
info: null,
code: 'GeneralError'
}
azure:service-bus:receiver:verbose ServiceBusError: DetachForcedError: The link 'G9:13642683:prd.gps-train.gps-tracker.messages/Subscriptions/processing-0f774707-92ad-a643-95b8-6e15eb539d7f' is force detached. Code: consumer(link35055). Details: InnerMessageReceiver was closed. TrackingId:638ebc4e000206d9000088ef62fbac03_G9_B32, SystemTracker:pzsbeotpr01:Topic:prd.gps-train.gps-tracker.messages|processing, Timestamp:2022-08-16T15:53:02
at translateServiceBusError (/node_modules/@groupeiot-itnovem/esncf-iot-servicebus/node_modules/@azure/service-bus/dist/index.js:4575:16)
at Receiver.StreamingReceiver._onAmqpError (/node_modules/@groupeiot-itnovem/esncf-iot-servicebus/node_modules/@azure/service-bus/dist/index.js:7082:33)
at Receiver.emit (node:events:527:28)
at emit (/node_modules/rhea-promise/dist/lib/util/utils.js:131:24)
at Object.emitEvent (/node_modules/rhea-promise/dist/lib/util/utils.js:142:9)
at Receiver.<anonymous> (/node_modules/rhea-promise/dist/lib/link.js:284:25)
at Receiver.emit (node:events:527:28)
at Receiver.link.dispatch (/node_modules/rhea/lib/link.js:59:37)
at Receiver.<anonymous> (/node_modules/dd-trace/packages/datadog-plugin-rhea/src/index.js:91:23)
at Receiver.link.on_detach (/node_modules/rhea/lib/link.js:162:32)

I am not sure is this is link to the sdk or not, I am a bit in the dark here, hence this issue. Any help would be greatly appreciated. I think the package could at least have some kind of re-connection mechanism to avoid this issue stopping the messages consumption.

[HarshaNalluru]

@laudibert, I’m checking with the service team on the force detached error. Can you provide us with the sample code to repro your issue, to help us investigate in terms of the reconnection?

[laudibert]

@HarshaNalluru any clue to what could be happening? Any chance of having an automatic re-connection?