redis-py is not compatible with select.select()

In a project involving websockets and redis, select.select() is being used to be notified when either a websocket or redis pub/sub has data ready for reading. For it to work at all, we have to peek under the covers of redis-py a bit to get at the socket’s file descriptor and receive the messages:

r = redis.StrictRedis()
pubsub = r.pubsub()

# subscribe to a few channels...

fd = pubsub.connection._sock.fileno()
while True:
    ready = select.select([fd], [], [])
    response = pubsub.parse_response()
    # do something with response

This generally works pretty well. However, we’ve discovered that when publishing multiple messages to subscribed channels in quick succession, select() tends to only return once for the first message, but not for subsequent messages. After a lot of digging, it turns out this is due to redis-py using buffered reads on its pub/sub socket. In particular, redis.connection.PythonParser.read uses readline, which is a buffered read operation. Several messages are read off the socket by the single call to readline, but all except the first message are buffered until the next time readline (or another read operation) is called. Due to this buffereing, select() will not fire again since there isn’t any additional data to be read off the socket.

It would be really powerful if redis-py could be made compatible with select.select(). As a proof of concept, I addressed the issue by monkey patching PythonParser’s read method to do unbuffered reads as shown below. This is clearly less efficient than doing buffered reads, but it seems something along these lines would be needed for select() compatibility.

# excerpt from redis.connection.PythonParser.read

...

# no length, read a full line
# readline is buffered, so #badnewsbears
#return self._fp.readline()[:-2]

# let's do it unbuffered instead!
buf = BytesIO()
byte = None
while byte != '\n':
    byte = self._fp.read(1)
    buf.write(byte)
buf.seek(0)
return buf.read()[:-2]

...

Note that this isn’t a valid solution by itself since hiredis also does buffered reads…

Does anyone have any thoughts on how redis-py could be adapted in a not-so-hacky way to make it play nice with select.select()?

[ntki]

Hi all!

Please consider changing from select.select() to select.poll(). If you have a long running app that handles reconnections and all that, the underlying filedescriptor number of the socket can be 1024>= and select could easily crash because of that. (in libhiredis too they changed that between 0.10-0.11)

Br

[andymccurdy]

redis-py 2.10 is out with the new pubsub system.

@ntki I’ve created a separate issue (#486) to track the change of moving from select.select() to something better.