Add support for clean teardown + reconstruction

Currently it seems that Rabbot is not able to reconnect succesfully, once it reaches retry limit. Whatever I’m trying to do (like calling .configure() again, calling .addConnection(), .addConnection() with new name, calling .retry(), .closeAll()+ all of the above, .shutdown() + all of the above, and so on), it just can’t reconnect cleanly. At best, I’m getting this (e.g. by calling rabbot.shutdown(); rabbot.configure( originalConfiguration ):

Potentially unhandled rejection [8] TypeError: Cannot read property 'define' of undefined
    at machina.Fsm.extend._define (.../app/node_modules/rabbot/src/exchangeFsm.js:43:17)
    at .<anonymous> (.../app/node_modules/rabbot/src/exchangeFsm.js:58:10)
    at Object.invokeSubscriber (.../app/node_modules/monologue.js/lib/monologue.js:181:19)
    at invoker (.../app/node_modules/monologue.js/lib/monologue.js:430:11)
    at arrayEach (.../app/node_modules/monologue.js/node_modules/lodash/index.js:1289:13)
    at Function.<anonymous> (.../app/node_modules/monologue.js/node_modules/lodash/index.js:3345:13)
    at Monologue.emit (.../app/node_modules/monologue.js/lib/monologue.js:439:6)
    at machina.Fsm.extend.states.acquired._onEnter (.../app/node_modules/rabbot/src/amqp/iomonad.js:196:11)
    at _.extend.transition (.../app/node_modules/machina/lib/machina.js:433:27)
    at Fsm.(anonymous function) [as transition] (.../app/node_modules/machina/lib/machina.js:580:60)
    at machina.Fsm.extend._onAcquisition (.../app/node_modules/rabbot/src/amqp/iomonad.js:108:9)
    at tryCatchReject (.../app/node_modules/when/lib/makePromise.js:845:30)
    at runContinuation1 (.../app/node_modules/when/lib/makePromise.js:804:4)
    at Fulfilled.when (.../app/node_modules/when/lib/makePromise.js:592:4)
    at Pending.run (.../app/node_modules/when/lib/makePromise.js:483:13)
    at Scheduler._drain (.../app/node_modules/when/lib/Scheduler.js:62:19)
Potentially unhandled rejection [9] TypeError: Cannot read property 'define' of undefined
    at machina.Fsm.extend._define (.../app/node_modules/rabbot/src/exchangeFsm.js:43:17)
    at .<anonymous> (.../app/node_modules/rabbot/src/exchangeFsm.js:58:10)
    at Object.invokeSubscriber (.../app/node_modules/monologue.js/lib/monologue.js:181:19)
    at invoker (.../app/node_modules/monologue.js/lib/monologue.js:430:11)
    at arrayEach (.../app/node_modules/monologue.js/node_modules/lodash/index.js:1289:13)
    at Function.<anonymous> (.../app/node_modules/monologue.js/node_modules/lodash/index.js:3345:13)
    at Monologue.emit (.../app/node_modules/monologue.js/lib/monologue.js:439:6)
    at machina.Fsm.extend.states.acquired._onEnter (.../app/node_modules/rabbot/src/amqp/iomonad.js:196:11)
    at _.extend.transition (.../app/node_modules/machina/lib/machina.js:433:27)
    at Fsm.(anonymous function) [as transition] (.../app/node_modules/machina/lib/machina.js:580:60)
    at machina.Fsm.extend._onAcquisition (.../app/node_modules/rabbot/src/amqp/iomonad.js:108:9)
    at tryCatchReject (.../app/node_modules/when/lib/makePromise.js:845:30)
    at runContinuation1 (.../app/node_modules/when/lib/makePromise.js:804:4)
    at Fulfilled.when (.../app/node_modules/when/lib/makePromise.js:592:4)
    at Pending.run (.../app/node_modules/when/lib/makePromise.js:483:13)
    at Scheduler._drain (.../app/node_modules/when/lib/Scheduler.js:62:19)

after which Rabbot claims that he has connection (connected is sent), but apparently is has no exchange configured, thus all publish calls fail.

I do not prefer to kill my whole process upon such failure. Rabbit is only small part of the whole, and it causing to shutdown whole world is little too inconvenient for me. Sure, alerting/monitoring should fire an alert to admin. Yes, all pending promises for publish should be rejected and not retried. Yes, I can create new connection from scratch. Yes, it will not fix the network. Restarting everything wont fix it, either, though. What I’d prefer to have is to have always-on, self-healing connection, based on best effort. I.e. if in given minute it cant deliver, fine, reject the promise. And it seems, that Rabbot has everything else I need. And I guess that this clean reconnect is like 99% there, too 😉

[Nopik]

Things started to work much better, when I added configureExchanges call in Topology.prototype.onReconnect

      return this.configureExchanges( this.definitions.exchanges, true )
        .then( function() {
             return this.configureBindings( this.definitions.bindings, true )
             ....
        }.bind( this ) );

and modified createExchange:

Topology.prototype.createExchange = function( options ) {
  this.definitions.exchanges[ options.name ] = options;
    return this.createPrimitive( Exchange, "exchange", options );
};

With those 2 small changes, when reconnection is done, both publisher and workers are working fine.

HOWEVER, shutting down is still not working fine, some old connection must be kept somewhere.

[arobson]

@TinOo512 @Nopik - I’ve managed to fix some issues related to reconnection / clean-up / shut-down as of 1.0.8 that I believe may have caused this. Would you try this out and re-open this issue if you’re still having the same problem?