Queued tasks (OrmQ) are not always acknowledged

Here’s our current config, we are using Django 2.2.16:

VERSION: 1.3.4
ACK_FAILURES: True
BULK: 1
CACHE: default
CACHED: False
CATCH_UP: True
COMPRESSED: False
CPU_AFFINITY: 0
DAEMONIZE_WORKERS: True
DISQUE_FASTACK: False
GUARD_CYCLE: 0.5
LABEL: Django Q
LOG_LEVEL: INFO
MAX_ATTEMPTS: 0
ORM: default
POLL: 0.2
PREFIX: DjangORM
QSIZE: True
QUEUE_LIMIT: 50
Q_STAT: django_q:DjangORM:cluster
RECYCLE: 500
REDIS: {}
RETRY: 2147483647
SAVE_LIMIT: 10000
SCHEDULER: True
SYNC: False
TESTING: False
TIMEOUT: 3300
WORKERS: 4

We have 3 clusters and 12 workers in total. QInfo:

QMonitor:

Sometimes queued tasks are not acknowledged and relevant Task instance (from the OrmQ payload) does not exist. This seems to happen mostly with tasks that have long execution time (200-400 seconds). TIMEOUT should be big enough and we get no errors from worker (we are using Sentry for error reporting). Any ideas? Oh and even though the SAVE_LIMIT is set to 10000, the limit doesn’t always hold i.e. it seems to sometimes ignore this part:

        if task["success"] and 0 < Conf.SAVE_LIMIT <= Success.objects.count():
            Success.objects.last().delete()

As you can see from qinfo, at the moment there are 10485 successful tasks in the database.

[Koed00]

@kennyhei already merged the limit fix, but need a bit more time to review the qmemory pr. Thanks for work, I’m sure you helped out a bunch of other people.

[kennyhei]

We haven’t encountered any problems after lowering the recycle setting so that memory is released more frequently. Going to close this issue.