No such file or directory: '/tmp/celery_once/ using file based backend

My tasks crash and do not run anymore due to a no such file error. I’m using the file based backend and this is my set up.

settings.py

CELERY_ONCE = { ‘backend’: ‘celery_once.backends.File’, ‘settings’: { ‘location’: ‘/tmp/celery_once/’, ‘default_timeout’: 60 * 60 } }

CELERY_BROKER_URL = ‘pyamqp://rabbitmq:5672’ CELERY_RESULT_BACKEND = ‘django-db’

CELERYD_HIJACK_ROOT_LOGGER = False

use json format for everything

CELERY_ACCEPT_CONTENT = [‘json’] CELERY_TASK_SERIALIZER = ‘json’ CELERY_TIMEZONE = ‘UTC’ CELERYBEAT_SCHEDULER = ‘django_celery_beat.schedulers:DatabaseScheduler’

CELERY_ONCE = { ‘backend’: ‘celery_once.backends.File’, ‘settings’: { ‘location’: ‘/tmp/celery_once/’, ‘default_timeout’: 60 * 60 } }

celery.py

from future import absolute_import import os

from celery import Celery from django.conf import settings

set the default Django settings module for the ‘celery’ program.

all = [ ‘celery’, ‘QueueOnce’, ] os.environ.setdefault(‘DJANGO_SETTINGS_MODULE’, ‘autobets.settings’) os.environ.setdefault(‘DJANGO_CONFIGURATION’, ‘Development’) import configurations

configurations.setup()

app = Celery(‘autobets’)

Using a string here means the worker will not have to

pickle the object when using Windows.

app.config_from_object(‘django.conf:settings’, namespace=‘CELERY’) app.conf.ONCE = settings.CELERY_ONCE app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

tasks.py

@shared_task(bind=True,base=QueueOnce, once={‘graceful’: True}) def get_events(self): do stuff bla bla bla

Before using celeryonce tasks would run as normal.

Do I need to create a tmp backend file first or does celery once create this?

stack trace.

[2020-02-13 20:11:43,505: INFO/MainProcess] mingle: searching for neighbors
[2020-02-13 20:11:44,810: INFO/MainProcess] mingle: all alone
[2020-02-13 20:11:44,865: WARNING/MainProcess] /usr/local/lib/python3.6/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2020-02-13 20:11:44,866: INFO/MainProcess] celery@9f2b7c337579 ready.
[2020-02-13 20:11:44,868: INFO/MainProcess] Received task: apimb.tasks.get_events[b423ad7e-cf2b-4137-8f2d-25bb934dc148]  
[2020-02-13 20:11:44,872: INFO/MainProcess] Received task: apimb.tasks.get_balance[6af923d7-f1a1-451a-bc8d-e4f07f3b2f06]  
[2020-02-13 20:11:44,876: INFO/MainProcess] Received task: apimb.tasks.get_orders[3c30ddee-22a2-4831-b256-0511ed1b8cc6]  
[2020-02-13 20:11:45,416: WARNING/ForkPoolWorker-1] login has been calledAPIClient
[2020-02-13 20:11:45,502: WARNING/ForkPoolWorker-1] GET Balance task has been called now
[2020-02-13 20:11:45,586: WARNING/ForkPoolWorker-2] login has been calledAPIClient
[2020-02-13 20:11:45,637: INFO/ForkPoolWorker-1] Task apimb.tasks.get_balance[6af923d7-f1a1-451a-bc8d-e4f07f3b2f06] succeeded in 0.6551041000020632s: None
[2020-02-13 20:11:45,652: WARNING/ForkPoolWorker-2] GET EVENTS task has been called now
[2020-02-13 20:11:45,706: WARNING/ForkPoolWorker-1] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:561: RuntimeWarning: Exception raised outside body: FileNotFoundError(2, 'No such file or directory'):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_52094e3196ca05f36eb013e2b749ce03'

  exc, exc_info.traceback)))
[2020-02-13 20:11:45,750: ERROR/ForkPoolWorker-1] Task apimb.tasks.get_balance[6af923d7-f1a1-451a-bc8d-e4f07f3b2f06] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_52094e3196ca05f36eb013e2b749ce03'
[2020-02-13 20:11:45,813: WARNING/ForkPoolWorker-1] GET ORDERS task has been called now
[2020-02-13 20:11:45,860: INFO/ForkPoolWorker-1] Task apimb.tasks.get_orders[3c30ddee-22a2-4831-b256-0511ed1b8cc6] succeeded in 0.10413699999844539s: None
[2020-02-13 20:11:45,875: WARNING/ForkPoolWorker-1] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:561: RuntimeWarning: Exception raised outside body: FileNotFoundError(2, 'No such file or directory'):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_55d76a4bebdad047aea38a8fd7e07c8e'

  exc, exc_info.traceback)))
[2020-02-13 20:11:45,937: ERROR/ForkPoolWorker-1] Task apimb.tasks.get_orders[3c30ddee-22a2-4831-b256-0511ed1b8cc6] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_55d76a4bebdad047aea38a8fd7e07c8e'
[2020-02-13 20:11:48,857: INFO/ForkPoolWorker-2] Task apimb.tasks.get_events[b423ad7e-cf2b-4137-8f2d-25bb934dc148] succeeded in 3.861357500001759s: None
[2020-02-13 20:11:48,878: WARNING/ForkPoolWorker-2] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:561: RuntimeWarning: Exception raised outside body: FileNotFoundError(2, 'No such file or directory'):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_f98116d19c287c5287360c9ef915d089'

  exc, exc_info.traceback)))
[2020-02-13 20:11:48,910: ERROR/ForkPoolWorker-2] Task apimb.tasks.get_events[b423ad7e-cf2b-4137-8f2d-25bb934dc148] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_f98116d19c287c5287360c9ef915d089'```

[cameronmaske]

My theory is that it’s due to the task still being in the broker (i.e. unprocessed) but the lock file is removed before the task is run. The lock is created before the task is sent into the broker.

If we break down the timeline, the following happens in order:

• Schedule task.
• Lock file is checked, and created if needs be. If no lock, the task is now in brokers (i.e. RabbitMQ) waiting to be picked up.
• Container/server restarts, or /tmp/ is cleared some other way.
• Lock file no longer exists, but the task is still in the broker.
• Task get picked up by worker, taken out of the broker.
• Task runs, goes to clean up lock file, but it no longer exists. Raises FileNotFoundError.

[xuhcc]

Yeah, that makes sense. Redis ignores the missing keys when DEL command is executed. So I think we should do the same with the file backend.