Track down requests timing issues

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn’t find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google “How to X in FastAPI” and didn’t find any information.
• I already read and followed all the tutorial in the docs and didn’t find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

None

Description

Hi, I have a setup with FastAPI and Celery (RabbitMQ+Redis) as background worker.

Some api are executed directly in FastAPI, and some others with heavy stuff are sent as Celery task.

Now, logically everything is working. But, when external systems start to make requests to my app (20-30 requests in parallel) the app is terribly slowed down, and I can’t figure out why.

I know it is impossible to help me without seeing the whole application (which is quite big and I don’t know how to narrow it down to a snippet), however I would like very much to have some hints to help me track down the issue.

Some facts:

• the server on which is running is a very good machine, so 20-30 requests is nothing, the loads is low
• when the app is slowed down, even the apis which use only fastapi are delayed: for example I have a trivial healthcheck endpoint which just returns 200, and it takes 90 seconds (!) to run
• exploring the apis which use celery, it seems like the task is received by celery several minutes later the api is called; so for example, I call the api at time 00:00:00, the api just runs the task (task.delay()) and waits, but in celery I see that the task is received at time 00:05:00, and lasts 10 seconds, so the overall time is 00:05:10 instead of 00:00:10
• from celery’s point of view, I can’t see any particular issue, the tasks are executed successfully and the run time is always approx 5-15 seconds
• so it seems like there is a delay between when fastapi receives the request, and when fastapi does execute the method associated with the request
• this same behavour, as I said, can be found as well in apis which only uses fastapi like the healthcheck, it seems again that fastapi takes approx 90 seconds from when it receives /healthcheck to when it runs the associated healthcheck() method

Now, how could I possibly debug all this to find out what is happening? Is there some kind of queue in apis requests so if there are N requests executing the N+1 is delayed?

Thank you

Operating System

Windows

Operating System Details

No response

FastAPI Version

0.63.0

Python Version

3.8.8

Additional Context

No response

[STeveShary]

@gygabyte017 Ok, I understand what you are asking.

The issue you are facing is the line: result = task.get(). This is a blocking method. It means that the python app will sit there until it is finished. My understanding is that you don’t want to “block”, but you also don’t want to return the result until the celery task is finished. In this case, you will want to “poll” the task periodically to understand if it is finished.

What that means is that you want to check the status, if not ready, then (async) sleep for a period of time, and then repeat. When the task is finished, you can call the task.get() function because you will know that it is ready to go!

Here is some code that I think would work (ish):

from celery.result import AsyncResult

@router.post("/test")
async def test():
    task = mytask.delay()
    while AsyncResult(task.id).status != 'SUCCESS':
        await asyncio.sleep(10)
    result = task.get()
    return result

This should allow you to run multiple background celery processes without blocking the FastAPI processes.

Things I used as a reference: https://testdriven.io/blog/fastapi-and-celery/#task-status https://docs.celeryproject.org/en/latest/reference/celery.result.html#module-celery.result https://docs.python.org/3/library/asyncio-task.html#sleeping

[sm-Fifteen]

Something that should probably get brought up more often in cases like these is that you can set Python in dev mode (or just set the PYTHONASYNCIODEBUG=1 environment variable if you’re not starting your server via a python script) to know when and where your async event loop gets blocked.