Does it make sense to use ThreadPoolExecutor in fastAPI

Hi All,

I have a super simple app that has only one endpoint. This endpoint loads data from a database and this is parallelised using a ThreadPoolExecutor. For example:

@app.get('/load_from_db')
def load_from_db():
    ....
    with concurrent.futures.ThreadPoolExecutor() as executor:
        for dataset in datasets:
            executor.submit(dataset.load_from_database)
    ....

Now I changed from Flask to FastAPI. I have declared my function as def load_from_db such that it is executed in a different thread-pool and does not block the main thread.

OUT-DATED As a result my service now is 10x slower than using Flasks? I tried to set max_workers=5 but did no really help. What is the reason?

EDIT: I created again some thorough test cases and it turns out fastAPI is not slower for me. Let’s change the question to:

Is it safe & does it make sense to use ThreadPoolExecutor in fastAPI?

[nzig]

@hellocoldworld is correct. I will add that in some cases you do actually need a thread pool, if you want to mix async with blocking operations. In that case you can use Starlette’s run_in_threadpool:

from starlette.concurrency import run_in_threadpool

@app.get('/')
async def():
   await do_something_async()
   await run_in_threadpool(do_something_blocking)

This is actually what FastAPI uses internally if you define your handler as def (and not async def).

[nzig]

Yes, it’s slightly slower. If you use ThreadPoolExecuter in a def function in FastAPI, what happens is:

1. Thread 1 runs FastAPI. Since FastAPI is an async application and your def function might block, it calls it with run_in_threadpool, which runs it in a thread 2.
2. Thread 2 runs your function. Since you created a ThreadPoolExecuter, this creates thread 3.
3. Thread 3 runs whatever you did with the ThreadPoolExecuter.

This means for the handler to complete you need 4 thread switches (1->2->3->2->1).

If you use an async def there are 0 thread switches, and if you use async def with run_in_threadpool there are 2. Since each thread switch adds overhead, using ThreadPoolExecuter inside a def function will probably be slower.