Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Configure call across containers in a docker stack not establishing provided topology, only default response

See original GitHub issue

See reproduction here: https://github.com/rhyslbw/rabbot-configure-docker

Context Two services in a docker-compose file, a rabbot client using the configure api to connect to a rabbitmq:3-management broker in the stack via the default network.

Problem When bringing up the stack via docker-compose up only the default response topology is created after the connection is established. When the client service is run independently on my dev machine with ports mapped into the broker, it connects and configures the defined topology as expected.

Notes

➜  rabbot-configure-docker git:(master) docker-compose up            
Starting rabbotconfiguredocker_broker_1 ... 
Starting rabbotconfiguredocker_broker_1 ... done
Starting rabbotconfiguredocker_client_1 ... 
Starting rabbotconfiguredocker_client_1 ... done
Attaching to rabbotconfiguredocker_broker_1, rabbotconfiguredocker_client_1
client_1  | 2017-09-30T03:53:23.191Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.223Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.224Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.226Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.233Z [rabbot.topology] Failed to create reply queue for connection name 'default Error: Failed to create queue '10c696ec7ae1.node.1.response.queue' on connection 'default' with 'No endpoints could be reached'
client_1  |     at Topology.<anonymous> (/usr/src/app/node_modules/rabbot/src/topology.js:196:10)
client_1  |     at onConnectionFailed (/usr/src/app/node_modules/rabbot/src/topology.js:208:13)
client_1  |     at /usr/src/app/node_modules/rabbot/src/topology.js:214:6
client_1  |     at Object.invokeSubscriber (/usr/src/app/node_modules/monologue.js/lib/monologue.js:181:19)
client_1  |     at invoker (/usr/src/app/node_modules/monologue.js/lib/monologue.js:430:11)
client_1  |     at /usr/src/app/node_modules/monologue.js/lib/monologue.js:324:5
client_1  |     at arrayEach (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:1289:13)
client_1  |     at Function.<anonymous> (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3345:13)
client_1  |     at /usr/src/app/node_modules/monologue.js/lib/monologue.js:436:7
client_1  |     at /usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3073:15
client_1  |     at baseForOwn (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:2046:14)
client_1  |     at /usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3043:18
client_1  |     at Function.<anonymous> (/usr/src/app/node_modules/monologue.js/node_modules/lodash/index.js:3346:13)
client_1  |     at fsm.emit (/usr/src/app/node_modules/monologue.js/lib/monologue.js:435:6)
client_1  |     at fsm.failed (/usr/src/app/node_modules/rabbot/src/connectionFsm.js:346:11)
client_1  |     at fsm.handle (/usr/src/app/node_modules/machina/lib/machina.js:613:25)
client_1  | No endpoints could be reached
client_1  | Done
client_1  | 2017-09-30T03:53:23.235Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.237Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.237Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.238Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.339Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.341Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.342Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.342Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.543Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.546Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.546Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.547Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:23.849Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:23.850Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:23.851Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:23.851Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:24 ===
broker_1  | Starting RabbitMQ 3.6.12 on Erlang 19.2.1
broker_1  | Copyright (C) 2007-2017 Pivotal Software, Inc.
broker_1  | Licensed under the MPL.  See http://www.rabbitmq.com/
broker_1  | 
broker_1  |               RabbitMQ 3.6.12. Copyright (C) 2007-2017 Pivotal Software, Inc.
broker_1  |   ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
broker_1  |   ##  ##
broker_1  |   ##########  Logs: tty
broker_1  |   ######  ##        tty
broker_1  |   ##########
broker_1  |               Starting broker...
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:24 ===
broker_1  | node           : rabbit@8ebcc31dee2b
broker_1  | home dir       : /var/lib/rabbitmq
broker_1  | config file(s) : /etc/rabbitmq/rabbitmq.config
broker_1  | cookie hash    : odpKFWjzmQ9v+NVANocB3w==
broker_1  | log            : tty
broker_1  | sasl log       : tty
broker_1  | database dir   : /var/lib/rabbitmq/mnesia/rabbit@8ebcc31dee2b
client_1  | 2017-09-30T03:53:24.253Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:24.255Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:24.255Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:24.255Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:24.761Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:24.762Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:24.762Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:24.763Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
client_1  | 2017-09-30T03:53:25.367Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:25.369Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:25.369Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:25.369Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Memory high watermark set to 799 MiB (838351257 bytes) of 1998 MiB (2095878144 bytes) total
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Enabling free disk space monitoring
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Disk free limit set to 50MB
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | Limiting to approx 1048476 file handles (943626 sockets)
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:25 ===
broker_1  | FHC read buffering:  OFF
broker_1  | FHC write buffering: ON
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Waiting for Mnesia tables for 30000 ms, 9 retries left
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Waiting for Mnesia tables for 30000 ms, 9 retries left
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Priority queues enabled, real BQ is rabbit_variable_queue
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Starting rabbit_node_monitor
client_1  | 2017-09-30T03:53:26.070Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | 2017-09-30T03:53:26.073Z [rabbot.connection] Failed to connect to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30) with, 'Error: connect ECONNREFUSED 172.20.0.2:5672'
client_1  | 2017-09-30T03:53:26.073Z [rabbot.connection] Cannot connect to `default` - all endpoints failed
client_1  | 2017-09-30T03:53:26.073Z [rabbot.io] Acquisition of connection 'default' failed with 'No endpoints could be reached'
client_1  | failed
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Management plugin: using rates mode 'basic'
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | msg_store_transient: using rabbit_msg_store_ets_index to provide index
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | msg_store_persistent: using rabbit_msg_store_ets_index to provide index
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | started TCP Listener on [::]:5672
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Management plugin started. Port: 15672
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Statistics database started.
broker_1  |  completed with 6 plugins.
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | Server startup complete; 6 plugins started.
broker_1  |  * rabbitmq_management
broker_1  |  * rabbitmq_web_dispatch
broker_1  |  * rabbitmq_management_agent
broker_1  |  * amqp_client
broker_1  |  * cowboy
broker_1  |  * cowlib
client_1  | 2017-09-30T03:53:26.876Z [rabbot.connection] Attempting connection to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | accepting AMQP connection <0.490.0> (172.20.0.3:54718 -> 172.20.0.2:5672)
broker_1  | 
broker_1  | =INFO REPORT==== 30-Sep-2017::03:53:26 ===
broker_1  | connection <0.490.0> (172.20.0.3:54718 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
client_1  | 2017-09-30T03:53:26.896Z [rabbot.connection] Connected to 'default' (amqp://guest:guest@broker:5672/%2f?heartbeat=30)
client_1  | connected
client_1  | 2017-09-30T03:53:26.910Z [rabbot.topology] Declaring queue '10c696ec7ae1.node.1.response.queue' on connection 'default' with the options: {"autoDelete":true,"subscribe":true,"uniqueName":"10c696ec7ae1.node.1.response.queue"}
client_1  | 2017-09-30T03:53:26.921Z [rabbot.topology] Declaring undefined exchange '' on connection 'default' with the options: {"passive":true}
client_1  | 2017-09-30T03:53:26.923Z [rabbot.queue] Starting subscription to queue '10c696ec7ae1.node.1.response.queue' on 'default'
client_1  | 2017-09-30T03:53:26.925Z [rabbot.queue] Subscription to (tracked) queue 10c696ec7ae1.node.1.response.queue - default started with consumer tag 10c696ec7ae1.node.1.response.queue

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:8 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
edorseycommented, Dec 21, 2017

I ran into this same problem as well. It appears that this module makes an assumption:

RabbitMQ will be running when rabbot tries to connect.

This is not the case with docker-compose.

To fix this, I did two things, after getting the connected event from rabbot. It is a bit hacky, but it appears to work:

let rabbit = require("rabbot")

let connectionConfig = {
  //connection details
};

rabbit.addConnection(connectionConfig) //Start by adding the connection only

rabbit.on("connected", function(a, b) {
  //Once we've connected, replace the connection promise with one that automatically resolves
  //Prior to this change, this is the original promise that failed. Publishing will always fail because of this.
  rabbit.connections[connectionConfig.name].promise = new Promise(function (resolve, reject) {
    resolve()
  })

  //Now reconfigure rabbit because if it fails its initial connection, the configuration doesn't ever get setup
  rabbit.configure({
    connection: connectionConfig,
    exchanges: [],
    queues: [],
    bindings:[]
  })
})

Not 100% that using the built-in Promise will cause any issues with the promises used in this project.

P.S. I looked at trying to fix this inside the module and submitting a PR, but I think it will require someone that better understands the internals to address. It might require a bit of re-working the internals from the bit of time I spent nailing down what was going on.

1reaction
antmarotcommented, Dec 3, 2018

I had the same issue.

It is true that more attempts to connect will be made but the promise returned by configure is rejecting on the first failed attempt.

I ended up doing this

const rabbot = require('rabbot');
const Promise = require('bluebird');

async function connect() {
  await rabbot.configure({
    connection: {
      /* connection details */
      retryLimit: 0, // no implicit retries
    },
    /* other options */
  });

  // Update internal promise - see @edorsey code
  const connectionName = 'default';
  rabbot.connections[connectionName].promise = Promise.resolve();
}

async function connectWithRetry(attempts = 1) {
  try {
    await connect();
  } catch (err) {
    if (attempts < 5) {
      console.log('Connection to RabbitMQ failed. Retrying in 5s...');
      await Promise.delay(5000);
      return connectWithRetry(attempts + 1);
    }
    throw err;
  }
}

try {
  await connectWithRetry();
} catch (err) {
  console.error(err);
  process.exit(1);
}
Read more comments on GitHub >

github_iconTop Results From Across the Web

Networking in Compose - Docker Documentation
Instead of just using the default app network, you can specify your own networks with the top-level networks key. This lets you create...
Read more >
Administer and maintain a swarm of Docker Engines
Administer and maintain a swarm of Docker Engines · Operate manager nodes in a swarm · Configure the manager to advertise...
Read more >
Deploy a stack to a swarm - Docker Documentation
To run through this tutorial, you need: A Docker Engine running in swarm mode. If you're not familiar with swarm mode, you might...
Read more >
Docker security
Of course, if the host system is setup accordingly, containers can interact with each other through their respective network interfaces — just like...
Read more >
docker stack deploy
Create and update a stack from a compose file on the swarm. This is a cluster management command, and must be executed on...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Rabbot internal logging not working when using rabbot.configure

Next page

Add support for clean teardown + reconstruction