Connections leak when restarting Huey.

Preamble Due to a memory leak inside our backend, we’ve developed a handy reload script to workaround the problem. By doing so we’ve come across a small problem when Huey is restarting.

The problem It seems that the consumer is not closing properly its connections and/or open files when restarting.

Here’s how to reproduce

• Start Huey

• Get process PID

• List all Redis open files ( 11 open files in this specific tests )

• Manually restart Huey by killing PID.

• List all Redis open files ( 22 in total )

• Manually restart Huey by killing PID ( again )

• List all Redis open files ( 33 in total )

Highly probable cause In consumer.py inside the run method, there’s this bit of code.

When restarting HUEY by using os.execl, the original process get overridden by the new instance of Huey and opened file handles (including open connections) are never properly closed (Most-likely because the PID is preserved)

Solution I will link a PR to what I think is a viable solution to this problems, Looking forward to your answer.

[coleifer]

I think I’ll mention this in the docs as a potential gotcha and close the issue.

A couple thoughts:

1. If using the consumer with multiprocessing then you won’t have issues, as the child processes exit and their resources are closed by the OS.
2. If you are using Python 3.4 or newer then you won’t have issues because Python opens files and sets the FD_CLOEXEC flag. The OS will handle closing the file-descriptors when an exec occurs.
3. If you are using Python 2.7 + threads + a process manager of some kind then you won’t have issues.

If you still want to implement some kind of cleanup routine, I’d suggest using fcntl.fcntl(fd, fcntl.FD_CLOEXEC) rather than os.close(fd). Letting the OS handle it seems like a much safer option.

[coleifer]

Well, I think the answer is: don’t use Python 2.7. I was reading up on fnctl and FD_CLOEXEC and ran across PEP 446, which:

This PEP proposes to make all file descriptors created by Python non-inheritable by default to reduce the risk of these issues.

It was accepted and is available from 3.4 onwards. I see from your logs that you’re using 2.7, so that explains the issue.

I verified that it behaves as advertised – running a threaded consumer and restarting it multiple times did not lead to a growth in the number of connections as reported by Redis.

To all the people who thumbs-up’d the issue – are you all running 2.7?