Calling broker.shutdown or subscription.cancel waits indefinitely if connection is not healthy

If connection to RabbitMQ is lost (reconnecting), calling broker.shutdown() or any of the subscription.cancel() waits indefinitely if connection is not healthy.

I’ve discovered this issue in a k8s environment. Basically the readiness probe of a service reported failure because the RabbitMQ connection is lost. After some time, k8s will restart the service, sending a SIGTERM signal. I am handling this signal to dispose resources, including shutting down the broker, but it hangs.

[cressie176]

Hi @satazor,

This is a good find, thank you for reporting. The underlying issue is with how rascal serializes subscriber channel operations. When a connection error occurs two things are happening

1. Rascal queues requests to obtain a new channel until the connection is re-established
2. The subscriber requests a channel in order to re-subscribe and becomes blocked

Additionally, when your code calls broker.shutdown…

1. Rascal iterates over all the subscriptions and cancels them. This involves getting the subscription’s channel.

Operations using the subscriptions channel are serialized via an in memory queue. (3) ends up being queued behind (2) which is blocked until the connection is re-established.

I’m not sure how easy it will be to detangle this, but will try to improve the behaviour if I can.

[cressie176]

Hi @satazor,

Just letting you know I haven’t forgotten about this issue. Sorry it’s taking so long.