Properly working function that uses pythonnet imported function stalls as a celery task.

Environment

• pythonnet 2.4.0.
• python 3.7.4
• OS: Void Linux

Details

I am working on a program that needs to use another dept.'s dotnet core dll’s and it works swimmingly when executed normally. For example, considering this code:

tasks.py:

import os
import sys
sys.path.append("./dll_lib")
import clr
clr.AddReference("System.IO")
import System.IO

from celery import Celery
app = Celery('Test', broker='amqp://guest:guest@localhost:5672//', backend="rpc://")

@app.task
def test(json_file):
    json_content = System.IO.File.ReadAllText(json_file)
    return 1

start.sh:

#!/bin/bash

CELERY_ALWAYS_EAGER=True
CELERYD_MAX_TASKS_PER_CHILD=1
CELERYD_CONCURRENCY=1
CELERYD_PREFETCH_MULTIPLIER=1
CELERY_RESULT_BACKEND="rpc://"
CELERY_RESULT_PERSISTENT=False

celery -A tasks worker --loglevel=INFO --max-tasks-per-child=1

When I test this task it runs fine as a standalone function. But this problem is nebulous, because I can issue a task with the path to a small json (<1mb) and it succeeds:

$ ipython -i
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tasks

In [2]: res = tasks.test.delay("/path/small.json")

In [3]: res.ready()
Out[3]: True

But when I do this with the path to a bigger json (~7mb), it stalls indefinitely:

In [4]: res_big = tasks.test.delay("/path/bigger.json")

In [5]: res_big.ready()
Out[5]: False

I have posted the same question to the Celery community and it is awful quiet over there. Does anybody have an idea what can be related to this issue ? It looks connected to memory used. This problem is not IO related, as I have tested it by reading a json as a string in the python way and using Newtonsoft.Json.dll to deserialize it. This test behaves along the similiarly with the same files, works with the small json, stalls on the bigger one.

Does anybody have a clue where the culprit may be hiding ? thx. S

[sanderboer]

This issue can be closed, I should have realised this type of unexpected irraltional behavior is a tell-tale sign of a lack of thread safety. I don’t know why I assumed celery would account for this, I should give it thread safe functions.

When I move the imports into the threaded function:

import os
import sys
sys.path.append("./dll_lib")

from celery import Celery
app = Celery('Test', broker='amqp://guest:guest@localhost:5672//', backend="rpc://")

@app.task
def test(json_file):
    import clr
    clr.AddReference("System.IO")
    import System.IO
    json_content = System.IO.File.ReadAllText(json_file)
    return 1

It works as expected…

[lostmsu]

This is weird. It sounds like there’s a problem when somebody initializes CLR from one thread and then calls File.ReadAllText from another. It should work.