question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to timeout when Sentinel Master is unreachable

See original GitHub issue

Is there a way to add a timeout to a get/set action? I’m looking for this in the docs and it’s either not there, or I’m somehow not seeing it.

I’m connecting ioredis to 3 sentinels using a master/slave setup. I can trigger a failover manually and everything works perfectly on the next request.

But, when I tell the current master to simulate an “unreachable” status: redis-cli -p 6379 DEBUG sleep 30

  • the sentinels notice and elect a new master
  • but the app waits for 30sec until the master is reachable again (now as a slave), then reconnects to the new sentinel master

My config is

{
    sentinels: [...the 3 sentinels],
    name: "mymaster",
    reconnectOnError: err => /^READONLY/.test(err.message),
}

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Reactions:2
  • Comments:5

github_iconTop GitHub Comments

2reactions
willrsterncommented, Dec 6, 2017

Also, here’s the Redis docs that show that sleep is a good way to simulate a master hanging. https://redis.io/topics/sentinel#testing-the-failover

0reactions
kapalkatcommented, Jun 4, 2020

So I have also followed described scenario and in my opinion this bug needs to be reopened. My setup: 3 sentinels, 3 redis slaves, 1 redis master. Sentinels env_vars: “REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS”: “10000”. I am simulating Redis master down with command: redis-cli -p 6379 DEBUG sleep 60. Whats going on is: After 10 seconds, sentinels know right a way about the failure and elects new master. ioredis does NOT detect that! No error is passed. ioredis is still connected to the old master and my app hangs… After 60 seconds when its back, ioredis detects that old master is not a master anymore and tries to reconnect to the new one with a success. The question is: why ioredis does not get new master from sentinels?

Read more comments on GitHub >

github_iconTop Results From Across the Web

High availability with Redis Sentinel
If two Sentinels agree at the same time about the master being unreachable, one of the two will try to start a failover....
Read more >
Configuring Redis servers for failover operation
If Redis. sentinel is configured and the master server is down, then one of the slave servers will be reconfigured as the master....
Read more >
3.4 Exercise - Sentinel Hands-on - Redis Developer Hub
sentinel failover-timeout - if a Sentinel voted another Sentinel for the failover of a given master, it will wait this many milliseconds to...
Read more >
Redis Sentinel Documentation
This delay is the failover-timeout you can configure in sentinel.conf . This means that Sentinels will not try to failover the same master...
Read more >
https://docs.sensu.io/sensu-core/latest/files/sent...
protected-mode no daemonize yes pidfile /var/run/redis/redis-sentinel.pid ... the master. sentinel parallel-syncs mymaster 1 # sentinel failover-timeout ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found