question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Connection retry doesn't always work after RabbitMQ is killed and restarted

See original GitHub issue

Or more accurately most times it doesn’t work.

Below is the broker config:

module.exports = {
  vhosts: {
    [config.RABBITMQ_VHOST]: {
      connection: {
        url: `${config.RABBITMQ_CONNECTION_STRING}`
      },
      exchanges: {
        [config.RABBITMQ_EXCHANGE_ENDPOINT_OUT]: {
          type: "topic",
          assert: false,
          check: true
        }
      },
      publications: {
        connectorPub: {
          exchange: `${config.RABBITMQ_EXCHANGE_ENDPOINT_OUT}`,
          routingKey: `${config.RABBITMQ_PUBLISH_ROUTING_KEY}`
        }
      }
    }
  }
};

I have 3 issues:

  1. Rascal doesn’t retry to connection if RabbitMQ has not been started (Rascal is started earlier than RabbitMQ). We mitigate this issue by exiting and let Kubernetes to restart the pods. Eventually Rascal will connect after RabbitMQ is started.
  2. After Rascal is connected to RabbitMQ, kill and restart RabbitMQ. Sometimes it will retry to connect, but most times it won’t. Connection reset error is provided though, but no retry. {“message”:“Broker error: Error: write ECONNRESET; vhost: /; connectionUrl: amqp://guest:***@localhost:31072”,“level”:“error”}
  3. If RabbitMQ is killed twice, Rascal will just give up. No retry has been observed.

I’m using the default retry configuration and I hope it should work as assumed. Please let me know whether this is a configuration issue or a bug.

Thanks.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:25 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
jliu-teracommented, Oct 2, 2020

Found the problem: vhost_initialized should be vhost_initialised. My bad, or spelling checker’s fault.

It all works as I expected now. Please close the ticket. Thanks a lot for the great library and your time on this issue.

0reactions
cressie176commented, Oct 2, 2020

Glad you found it. I’ve done a very similar things with the HTTP “Authorization” header.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Consumer doesn't consume messages after RabbitMQ restart
Consumer always reconnects to queue. Actual Behavior. Consumer doesn't reconnect to queue. At channel section in RabbitMQ management message: ".
Read more >
Detecting Dead TCP Connections with Heartbeats ... - RabbitMQ
Deactivating heartbeats is not recommended unless the environment is known to use TCP keepalives on every host (both RabbitMQ nodes and applications).
Read more >
RabbitMQ reconnect logic is not working on abrupt restart of ...
A couple of tests with a local instance of RabbitMQ showed that a connection.on('close') listener is always being fired when RabbitMQ ...
Read more >
RabbitMq Shovel does not restart - Google Groups
We're using RabbitMQ on the vessel with the Shovel plugin which transfers those messages from the vessel to an Azure ServiceBus in the...
Read more >
A tale of retries using RabbitMQ - Medium
We can set up the Microservice inbound queue with a x-dead-letter-exchange header and simply reject the message without re-enqueuing. RabbitMQ ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found