Django-q calls task twice or more

My background process is called twice (or more) but I’m really sure that should not be happening. My settings for Django Q:

Q_CLUSTER = {
    'name': 'cc',
    'recyle': 10,
    'retry': -1,
    'workers': 2,
    'save_limit': 0,
    'orm': 'default'
}

My test task function:

def task_test_function(email, user):
    print('test')

calling it from the commandline:

> python manage.py shell
>>> from django_q.tasks import async
>>> async('task_test_function', 'email', 'user')
'9a0ba6b8bcd94dc1bc129e3d6857b5ee'

Starting qcluster (after that I called the async)

> python manage.py qcluster
13:48:08 [Q] INFO Q Cluster-33552 starting.
...
13:48:08 [Q] INFO Q Cluster-33552 running.
13:48:34 [Q] INFO Process-1:2 processing [mobile-utah-august-indigo]
test
13:48:34 [Q] INFO Process-1:1 processing [mobile-utah-august-indigo]
test
13:48:34 [Q] INFO Processed [mobile-utah-august-indigo]
13:48:34 [Q] INFO Processed [mobile-utah-august-indigo]
...

And the function is called twice… For most functions I wouldn’t really care if they run twice (or more) but I have a task that calls send_mail and people that are invited receive 2 or more mails…

Is this a bug in Django Q or in my logic?

[cyliang]

I’ve been facing similar problem that some of my long tasks are being processed again and again even successfully processed.

I eventually resolved this by configuring a longer retry time.

According to http://django-q.readthedocs.io/en/latest/configure.html#retry, retry is

The number of seconds a broker will wait for a cluster to finish a task, before it’s presented again.

So I think that’s why my tasks are pushed into the queue again after 60 seconds (the default retry time) of unfinished execution and processed repeatedly

I’m not sure if this can help you but I think there might be someone like me being confused by similar problem.

Sorry for my poor English.

[jordanmkoncz]

I was having the same problem as you @Eagllus. I had a task which would run for about 180 seconds, so I had configured a timeout of 600 seconds to accommodate this task, but I still had my retry configured to 120 seconds (like it’s configured in the ORM example at http://django-q.readthedocs.io/en/latest/configure.html#orm). As a result, my task would get picked up by a worker, then 120 seconds later the same task would be picked up by a second worker, and then both workers would successfully process the task (in my case this means generating a report twice and sending it by email twice, which is obviously not what I want).

Configuring my retry to be longer than my timeout (I made it 60 seconds longer) resolved the issue for me too. I don’t feel like it’s obvious from the docs right now that django-q will behave the way it does if you have your retry configured to be shorter than your timeout and have a task which will take longer to process than your retry time. If someone hadn’t already read through the comments on this issue, I feel like it would be easy to misinterpret the docs for the retry setting.

I’m struggling to imagine a scenario where you’d actually want to have retry configured to be shorter than your timeout and get the duplicate task processing behaviour that occurs as a result. I assume there is a scenario where this would make sense, but it seems like for the majority of use cases you’d want to make sure your retry is longer than your timeout so that you can avoid this duplicate task processing behaviour. Maybe the docs for the retry setting could be updated to explain all of this, i.e. explain how django-q will behave when retry is shorter or longer than timeout, and recommend configuring it to be longer for most use cases.