Question - handling connection closure with reason "shutdown"

Hey @cressie176,

I have an application consuming a queue. Currently, when there is a maintenance of our RMQ cluster (and I guess that RMQ node is stopped and replaced or restarted). My application gets "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'" message and it stops consuming the queue until I manually restart the application. Within the application I initialize its subscription to queue in following way:

const broker = await Broker.create(...);

broker.on("error",
  err => logger.error("RabbitBroker (rascal) - error", err)
);

const subscription = await broker.subscribe(...);
subscription
  .on("message", (message, content, ackOrNack) => {
    ...
  })
  .on("error",
    err => logger.error("Subscription error", err)
  );

During last maintenance, it logged those two messages (emitting an error events for both subscription and broker):

ERROR Subscription error - Error: Connection closed: 320 (CONNECTION-FORCED) with message "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"

ERROR RabbitBroker (rascal) - error - Error: Connection closed: 320 (CONNECTION-FORCED) with message "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"

When an error event is emitted for broker or subscription is there any chance that it will recover? Can my app detect somehow that connection won’t be restored/rebuild in this case? I am thinking about terminating the application and letting underlying (container) orchestrator to restart it - but I am afraid there could be other error event types where restarting could be harmful.

Thank you for making this awesome library and supporting its users!

[cressie176]

Hi @matej-prokop,

Thanks for the extra debug. It shows that the connection was successfully re-established, but that attempts to borrow channels from the pool never yield a result. You can see the queue growing (1, 2, 3) but never released - available is 0 and borrowed is zero.

It’s possible that there’s a bug somewhere and channel allocation, which is paused while the connection is unavailable, is somehow never resumed when it recovers. It could also be that the channel pool has somehow choked. I’ll see if I can reproduce.

[matej-prokop]

All my tests were done with rascal 13.0.2 and amqplib 0.7.1 so it must be something else. I will retry with DEBUG=rascal:* and share output here.