Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Redis + sentinel connection infinite loop

See original GitHub issue

Hi there. Here are the basics:

redis-py ver 3.2.0 with a redis ver 5.0.3
python 2.7.5 on centos 7
nginx + uwsgi When uwsgi is started, i define the redis connections and quickly do a health check (write a key, read it back). That works fine. However, the first redis operation resulted from an http request to the server throws the driver into a continuous connection reattempt loop. See below. Note the connection is going via a Sentinel driver. Downgrading to redis-py 2.10.6 fixed the issue for now.

Any thoughts much appreciated!

  File "/usr/lib/python2.7/site-packages/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/site-packages/bottle.py", line 1740, in wrapper
    rv = callback(*a, **ka)
  File "./models/agent.py", line 26, in request_wrapper
    d = rdb.hgetall("some_key_name")
  File "/usr/lib/python2.7/site-packages/redis/client.py", line 2649, in hgetall
    return self.execute_command('HGETALL', name)
  File "/usr/lib/python2.7/site-packages/redis/client.py", line 772, in execute_command
    connection = pool.get_connection(command_name, **options)
  File "/usr/lib/python2.7/site-packages/redis/connection.py", line 986, in get_connection
    self._checkpid()
  File "/usr/lib/python2.7/site-packages/redis/sentinel.py", line 130, in _checkpid
    self.disconnect()
  File "/usr/lib/python2.7/site-packages/redis/connection.py", line 1037, in disconnect
    self._checkpid()
  File "/usr/lib/python2.7/site-packages/redis/sentinel.py", line 130, in _checkpid
    self.disconnect()

..................

  File "/usr/lib/python2.7/site-packages/redis/connection.py", line 1037, in disconnect
    self._checkpid()
  File "/usr/lib/python2.7/site-packages/redis/sentinel.py", line 130, in _checkpid
    self.disconnect()
  File "/usr/lib/python2.7/site-packages/redis/connection.py", line 1037, in disconnect
    self._checkpid()
RuntimeError: maximum recursion depth exceeded

Issue Analytics

State:
Created 5 years ago
Reactions:2
Comments:8 (5 by maintainers)

Top GitHub Comments

5reactions

andymccurdycommented, Mar 15, 2019

3.2.1 is out with this fix.

2reactions

whitehatboxercommented, Mar 8, 2019

In addition to it.

I think the problem was caused when using os.fork to fork a child which will conenct to redis in redis sentinel mode.

such as

And I check code, as you can see in the pic below.

Obviously, it was caused by this code.

When os.getpid() get different value, error happen

Top Results From Across the Web

Redis + sentinel connection infinite loop · Issue #1144 - GitHub

When uwsgi is started, i define the redis connections and quickly do a health check (write a key, read it back). That works...

Redis master slave sync infinite loop - Stack Overflow

Our slave master replication keeps killing our redis connection. It tries to connect to the master and replicate the master data. Unfortunately the...

The Endless Redis Replication Loop: What, Why and How to ...

Redis ' replication is an indispensable tool – it can be used both to heighten the availability of your Redis setup (you can...

Redis Node moving out of cluster repeatedly - Madhur Ahuja

Recently, we noticed a production issue where some of the redis slave nodes were repeatedly being kicked out of the cluster.

High availability with Redis Sentinel

Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a replica is promoted to master,...