[QUESTION] Best way to handle multiple incoming requests that trigger external APIs

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn’t find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google “How to X in FastAPI” and didn’t find any information.
• I already read and followed all the tutorial in the docs and didn’t find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from fastapi import FastAPI
import asyncio


app = FastAPI()

async def expensive_request():
    asyncio.sleep(10)
    return {"foo": "bar"}

@app.get("/request")
async def request():
    result = await expensive_request() 
    return {"message": result}

Description

The expensive_request method calls an external API service that is rate limited. When multiple users call the /request endpoint at the same time, the expensive_request gets triggered several times. What are the ways to group these multiple requests into one awaited one?

Operating System

Linux

Operating System Details

No response

FastAPI Version

0.65.2

Python Version

3.9.9

Additional Context

No response

[juangaitanv]

One way, I found easy to implement, to wait for already pending requests (expensive_request) is to use a cache in the following way:

import asyncio
from fastapi import FastAPI

EXPENSIVE_REQUEST_KEY = "expensive"
tasks_cache = {}

app = FastAPI()


async def coro_wrapper(result):
    return result


async def expensive_request():
    print("Starting Expensive request...")
    await asyncio.sleep(5)
    print("Finished Expensive request...")
    return {"foo": "bar"}


@app.get("/request")
async def request():
    global tasks_cache

    future = None
    if task := tasks_cache.get(EXPENSIVE_REQUEST_KEY):
        if task.done():
            future = coro_wrapper(task.result())
        else:
            future = task
    else:
        tasks_cache[EXPENSIVE_REQUEST_KEY] = asyncio.create_task(expensive_request())
        future = tasks_cache[EXPENSIVE_REQUEST_KEY]

    result = await future
    tasks_cache = {}  # Reset cache
    return {"message": result}

What are your thoughts on this implementation 🙏 ?

[DavidSantacruzR]

@juangaitanv well not exactly like that. In other words, you might want to use a child_process from the Node packages (https://nodejs.org/api/child_process.html) and using this to call the script where you have the function or the python implementation.