Messages to publish are lost when RabbitMQ connection fails

I have this simple publication code:

const broker = await Rascal.BrokerAsPromised.create(Rascal.withDefaultConfig(config));

broker.on('error', (error) => {
    console.error(`Broker error: ${error.message}`);
});

setInterval(async function() {
      try {
        const publication = await broker.publish('pub', 'hello world');
        publication.on('error', (error) => {
            console.error(`Publication error: ${error.message}`);
        });
      } catch (error) {
        console.error(`Rascal config error: ${error.message}`);
      }
}, 1000);

It’s running fine until I shutdown the RabbitMQ server. Then I see this error in the logs:

Broker error: Connection closed: 320 (CONNECTION-FORCED) with message "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"

The messages are not published now, obviously. There is no Publication error in the logs though, which caused some confusion for me.

When I start the RabbitMQ server, I’d expect the in-meantime messages to be published after Rascal reconnects. This doesn’t happen, the messages are lost. If I understand the code correctly they’re queued in poolQueue: https://github.com/guidesmiths/rascal/blob/a68834d46ccc0b476d4530a21368ca2b234348d6/lib/amqp/Vhost.js#L211 but then after reconnection the queue is overridden with an empty one in https://github.com/guidesmiths/rascal/blob/a68834d46ccc0b476d4530a21368ca2b234348d6/lib/amqp/Vhost.js#L245

My workaround is to add a local queue and pause/resume it based on the connection status:

const broker = await Rascal.BrokerAsPromised.create(Rascal.withDefaultConfig(config));

const messagesToPublishQueue = async.queue((_, next) => {
	next();
}, 1);

messagesToPublishQueue.pause();

broker
    .on('error', (error) => {
        console.error(`Broker error: ${error.message}`);
    })
	.on('connect', () => {
		messagesToPublishQueue.resume();
	})
	.on('disconnect', () => {
		messagesToPublishQueue.pause();
	});

setInterval(async function() {
    messagesToPublishQueue.push(null, async () => {
      try {
        const publication = await broker.publish('pub', 'hello world');
        publication.on('error', (error) => {
            console.error(`Publication error: ${error.message}`);
        });
      } catch (error) {
        console.error(`Rascal config error: ${error.message}`);
      }
    });
}, 1000);

[cressie176]

Thanks for explaining again @rogatty, I understand the issue now. Agree it’s a problem that needs fixing.

[rogatty]

Understood. It’s up to you if and how you want to document the current behaviour 🙂 If someone tries to debug what’s happening they’ll probably end up here and can learn from the discussion above.

Thanks again for fixing the issue so quickly!