Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Connections leak when restarting Huey.

See original GitHub issue

Issue Description

Preamble Due to a memory leak inside our backend, we’ve developed a handy reload script to workaround the problem. By doing so we’ve come across a small problem when Huey is restarting.

The problem It seems that the consumer is not closing properly its connections and/or open files when restarting.

Here’s how to reproduce

  • Start Huey image

  • Get process PID image

  • List all Redis open files ( 11 open files in this specific tests ) image

  • Manually restart Huey by killing PID. image

  • List all Redis open files ( 22 in total ) image

  • Manually restart Huey by killing PID ( again ) image

  • List all Redis open files ( 33 in total ) image

Highly probable cause In inside the run method, there’s this bit of code. image

When restarting HUEY by using os.execl, the original process get overridden by the new instance of Huey and opened file handles (including open connections) are never properly closed (Most-likely because the PID is preserved)

Solution I will link a PR to what I think is a viable solution to this problems, Looking forward to your answer.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:5
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

coleifercommented, Nov 20, 2018

I think I’ll mention this in the docs as a potential gotcha and close the issue.

A couple thoughts:

  1. If using the consumer with multiprocessing then you won’t have issues, as the child processes exit and their resources are closed by the OS.
  2. If you are using Python 3.4 or newer then you won’t have issues because Python opens files and sets the FD_CLOEXEC flag. The OS will handle closing the file-descriptors when an exec occurs.
  3. If you are using Python 2.7 + threads + a process manager of some kind then you won’t have issues.

If you still want to implement some kind of cleanup routine, I’d suggest using fcntl.fcntl(fd, fcntl.FD_CLOEXEC) rather than os.close(fd). Letting the OS handle it seems like a much safer option.

coleifercommented, Nov 20, 2018

Well, I think the answer is: don’t use Python 2.7. I was reading up on fnctl and FD_CLOEXEC and ran across PEP 446, which:

This PEP proposes to make all file descriptors created by Python non-inheritable by default to reduce the risk of these issues.

It was accepted and is available from 3.4 onwards. I see from your logs that you’re using 2.7, so that explains the issue.

I verified that it behaves as advertised – running a threaded consumer and restarting it multiple times did not lead to a growth in the number of connections as reported by Redis.

To all the people who thumbs-up’d the issue – are you all running 2.7?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Consuming Tasks — huey 2.4.4 documentation - Read the Docs
I use supervisord to manage my huey process, so I set the directory to the root of my site ... ability to terminate...
Read more >
huey Documentation - Read the Docs
When the consumer receives the hang-up signal, any tasks being executed will be allowed to finish before the restart occurs. Note: If you...
Read more >
Huey | Vornado
Replace Water Bottle and inspect the base of Huey for leaks. Leaking? In the slight chance your Huey has leaks, contact Consumer Services...
Read more >
A connection leak is seen following failover and reconnection to a queue manager if subscriptions are used from the resource.
Read more >
Run huey task queue in the background with supervisor on ...
... Create a huey.conf file to be loaded by supervisor ... -c /usr/local/etc/supervisord.conf restart huey commands: 01_start_supervisor: ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found