Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Connections leak when restarting Huey.

See original GitHub issue

Issue Description

Preamble Due to a memory leak inside our backend, we’ve developed a handy reload script to workaround the problem. By doing so we’ve come across a small problem when Huey is restarting.

The problem It seems that the consumer is not closing properly its connections and/or open files when restarting.

Here’s how to reproduce

Start Huey
Get process PID
List all Redis open files ( 11 open files in this specific tests )
Manually restart Huey by killing PID.
List all Redis open files ( 22 in total )
Manually restart Huey by killing PID ( again )
List all Redis open files ( 33 in total )

Highly probable cause In consumer.py inside the run method, there’s this bit of code.

When restarting HUEY by using os.execl, the original process get overridden by the new instance of Huey and opened file handles (including open connections) are never properly closed (Most-likely because the PID is preserved)

Solution I will link a PR to what I think is a viable solution to this problems, Looking forward to your answer.

Issue Analytics

State:
Created 4 years ago
Reactions:5
Comments:6 (4 by maintainers)

Top GitHub Comments

2reactions

coleifercommented, Nov 20, 2018

I think I’ll mention this in the docs as a potential gotcha and close the issue.

A couple thoughts:

If using the consumer with multiprocessing then you won’t have issues, as the child processes exit and their resources are closed by the OS.
If you are using Python 3.4 or newer then you won’t have issues because Python opens files and sets the FD_CLOEXEC flag. The OS will handle closing the file-descriptors when an exec occurs.
If you are using Python 2.7 + threads + a process manager of some kind then you won’t have issues.

If you still want to implement some kind of cleanup routine, I’d suggest using fcntl.fcntl(fd, fcntl.FD_CLOEXEC) rather than os.close(fd). Letting the OS handle it seems like a much safer option.

1reaction

coleifercommented, Nov 20, 2018

Well, I think the answer is: don’t use Python 2.7. I was reading up on fnctl and FD_CLOEXEC and ran across PEP 446, which:

This PEP proposes to make all file descriptors created by Python non-inheritable by default to reduce the risk of these issues.

It was accepted and is available from 3.4 onwards. I see from your logs that you’re using 2.7, so that explains the issue.

I verified that it behaves as advertised – running a threaded consumer and restarting it multiple times did not lead to a growth in the number of connections as reported by Redis.

To all the people who thumbs-up’d the issue – are you all running 2.7?