Lightrun
question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Documentation about greenlets with djhuey

See original GitHub issue

Issue Description

Hi,

When trying to make use of greenlets workers, the consumer task just doesn’t work. I tried to monkey patch using gevent, but it seems the worker thread is not able to connect to DB.

Settings

HUEY = {
    'connection': {'host': self.HUEY_HOST, 'port': self.HUEY_PORT},
}

Additional run_huey_greenlet management command :

from huey.contrib.djhuey.management.commands.run_huey import Command as RunHueyCommand
from django.conf import settings


class Command(RunHueyCommand):

    def __init__(self, *args, **kwargs):
        if settings.HUEY.get('consumer', {}).get('worker_type') == 'greenlet':
            from gevent import monkey
            monkey.patch_all()
        super(Command, self).__init__(*args, **kwargs)

CLI

./manage.py run_huey_greenlet

DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 139937579968256 and this is thread id 139937120251248.

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
antwancommented, Mar 31, 2017

It worked when patching the manage.py bootstrap script, as this is where the internal DB connection is made. Patching the actual command file doesn’t work as it’s too late.

#!/usr/bin/env python
import os
import sys

if 'run_huey' in sys.argv:
    from gevent import monkey
    monkey.patch_all()

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "conf")
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)

I think it’s worth mentioning that in the doc + troubleshooting, because :

  1. It’s not optional as stated, it’s a mandatory thing to do.
  2. Most of the people use djhuey and there is no mention anywhere about how to do it properly using django.
0reactions
coleifercommented, Oct 2, 2017
Read more comments on GitHub >

github_iconTop Results From Across the Web

Huey Extensions — huey 2.4.4 documentation - Read the Docs
The MiniHuey consumer runs inside a greenlet in your main application process. ... 'huey.contrib.djhuey', # Add this to the list. # ... )....
Read more >
Low-level utilities — gevent 22.10.3.dev0 documentation
Represents a tree of greenlets. In gevent, the parent of a greenlet is usually the hub, so this tree is primarily arganized along...
Read more >
[Django] #33092: PyMemcacheCache backend fails when ...
suggested in the Django documentation at: ... in gunicorn with the gevent worker class. ... by another greenlet: <bound method Waiter.switch of <gevent....
Read more >
python 任务队列huey_huey：小型多线程任务队列 - CSDN Blog
multi-process, multi-thread or greenlet task execution models. schedule tasks to execute at a given time, or after a given delay.
Read more >
Python 2.4.3 Multi Threading Change Number Of Concurrency ...
This package is an extension of huey contrib djhuey package that allows users ... read the documentation.huey supports: multiprocess multithread or greenlet ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found