pubsub.run_in_thread : 'sleep_time' and documentation

The pubsub.run_in_thread implementation works great for us, very nice work indeed.

We do however have a question about the parameter name ‘sleep_time’ plus its default value, and about the documentation.

Last week, we had a production issue, which was entirely our fault - not tested thoroughly enough. An example from the documentation had been taken too literally, and our CPU cost estimate was wrong.

Note: where I say ‘documentation’, it may fall under your control or maybe not. We also collected info from StackOverflow and such, we are not entirely sure where the examples came from 😉

We already solved all issues, but recommend the following:

Change sleep_time to socket_timeout. The name sleep_time was very confusing to us, since it’s no sleep at all. It’s used as a TCP timeout value in the select() statement, where control is returned to the OS.

It’s initial value is 0. This is very CPU unfriendly. The documentation says ‘be nice to the CPU’ by setting it to 0.001. This is a tiny bit more nice than 0, but turned out to be not nice enough for our scenario. On a live server, the run_in_thread was running in hundreds of processes on one server, each causing 0.5-1.0% CPU, clogging everything. Revert and reboot needed (and alas, an hour of production downtime due to dependencies).

The initial value for sleep_time (socket_timeout) should be None in our opinion. Since this week, we run the pubsub as a daemon thread with sleep_time=None, and after initialization, this takes 0.0% CPU. It hands over control to the OS. Our hundreds of processes on that live server run fine now.

Caveat: You should not try to use this thread for pubsub connections to more than 1 redis instance. It’s better to fire up an extra thread to accomplish that. Also see the python docs on select: https://docs.python.org/2/howto/sockets.html . Excerpt: give it a nice long timeout (say a minute) unless you have good reason to do otherwise

Kind regards, TW

[andymccurdy]

Thanks for the feedback. I’ll add this to the redis-py 4.0 TODO list. FYI, 4.0 will be Python 3.5+ only.

[chayim]

@sav-norem. Wanna?