Connections leak when restarting Huey.See original GitHub issue
Preamble Due to a memory leak inside our backend, we’ve developed a handy reload script to workaround the problem. By doing so we’ve come across a small problem when Huey is restarting.
The problem It seems that the consumer is not closing properly its connections and/or open files when restarting.
Here’s how to reproduce
Get process PID
List all Redis open files ( 11 open files in this specific tests )
Manually restart Huey by killing PID.
List all Redis open files ( 22 in total )
Manually restart Huey by killing PID ( again )
List all Redis open files ( 33 in total )
Highly probable cause
consumer.py inside the
run method, there’s this bit of code.
When restarting HUEY by using
os.execl, the original process get overridden by the new instance of Huey and opened file handles (including open connections) are never properly closed (Most-likely because the PID is preserved)
Solution I will link a PR to what I think is a viable solution to this problems, Looking forward to your answer.
- Created 4 years ago
- Comments:6 (4 by maintainers)
Top GitHub Comments
I think I’ll mention this in the docs as a potential gotcha and close the issue.
A couple thoughts:
If you still want to implement some kind of cleanup routine, I’d suggest using
fcntl.fcntl(fd, fcntl.FD_CLOEXEC)rather than
os.close(fd). Letting the OS handle it seems like a much safer option.
Well, I think the answer is: don’t use Python 2.7. I was reading up on fnctl and FD_CLOEXEC and ran across PEP 446, which:
It was accepted and is available from 3.4 onwards. I see from your logs that you’re using 2.7, so that explains the issue.
I verified that it behaves as advertised – running a threaded consumer and restarting it multiple times did not lead to a growth in the number of connections as reported by Redis.
To all the people who thumbs-up’d the issue – are you all running 2.7?