Smoke Detector instances can't coordinate without MS
See original GitHub issueWhat problem has occurred? What issues has it caused?
MS has been down this morning. All online Smoke Detector instances (iBug, ArtOfCode, teward/Solar Flare, teward/Lunar Eclipse) keep claiming they’re receiving failover signals and starting up. In the past four hours of MS being down, about 26 standby requests have been made in Charcoal HQ (see transcript here and the next time period, ctrl+f for !!/standby
).
What would you like to happen/not happen?
Metasmoke instances should coordinate between themselves over chat somehow:
- which ones are online
- which one is the primary instance
- whether the primary instance is healthy and responsive
We should never have two instances of metasmoke running and responding simultaneously, even when MS is down.
We should never have more than one instance running actively (i.e. out of standby). All other instances beyond the primary should be in standby.
We should never have multiple instances repeatedly claiming to have received a failover signal and leaving standby while another instance is already primary and active. This should be maintained even when MS is down.
When no instances are present and healthy and online asserting primacy, other instances should attempt to assert primacy. If multiple assert primacy simultaneously there should be a means to allow them to decide who “wins”: for example, if they all post to Charcoal HQ asserting primacy, the one with the earlier chat message ID wins and becomes the primary, others do not leave standby because they recognise they had a later chat message ID.
Issue Analytics
- State:
- Created 5 years ago
- Comments:5 (4 by maintainers)
I have a neat little websocket server around somewhere, and plenty of machine space to host it. I’d suggest using that - all instances, including standbys, connect to it, and essentially use it as a relay between themselves to co-ordinate when MS is down.
Does sound interesting, though - we already know how to talk and listen to chat, could be neat to make a new room where Smokey talks to itself. Would need to be low volume, but it’d completely remove us from the hosting question for this