How to pass correlation_id to tasks executed in a multithreaded environment?

EDIT: Changed the name of the issue for better searchability; you can find the solution to the question here

---

Hey there!

I feel pretty stupid asking this question, but can you explain to me how I should create my logger instance to have a correlation_id?

Currently I create my logger at the top of the a router file:

import logging
from fastapi import APIRouter, HTTPException

LOG = logging.getLogger(__name__)

router = APIRouter(prefix="/my/route", responses={404: {"description": "Not found"}})

@router.get("/")
def handler():
   LOG.info("Hello!")

And I get

[2022-07-19T20:37:48] INFO [None] path.to.module | Hello

when my logging configuration is as follows:

    "formatters": {
        "default": {
            "format": "[%(asctime)s] %(levelname)s [%(correlation_id)s] %(name)s | %(message)s",
            "datefmt": "%Y-%m-%dT%H:%M:%S",
        }
    },

    app.add_middleware(
        CorrelationIdMiddleware,
        header_name='X-Request-ID',
        generator=lambda: uuid4().hex,
        validator=is_valid_uuid4,
        transformer=lambda a: a,
    )
 

– I would like to have my correlation_id show up in my log like so:

[2022-07-19T20:37:48] INFO [8fe9728a] path.to.module | Hello

I can’t get anything about it in both the Starlette and FastAPI documentation. It’s like everybody knows this and it’s not worth mentionning 🤔

Can you give me an example of how I should get a logger instance to have the request id show up?

Thanks for your help!

[philippefutureboy]

Thanks for the additional info Jonas! That was enough to solve the issue 👌

I took the solution presented in this medium article as a way to transfer the contextvars.

Here’s my final impl for the run_all ThreadPoolExecutor function:

def run_all(
    tasks: Dict[str, TaskTuple],
    pass_contextvars: bool = False,
    max_workers: int = 20,
):
    exec_kwargs = {}
    if pass_contextvars:
        parent_context = contextvars.copy_context()
        exec_kwargs = {"initializer": _set_context, "initargs": (parent_context,)}

    with ThreadPoolExecutor(max_workers=max_workers, **exec_kwargs) as executor:
        jobs = {}
        for name, task in tasks.items():
            fn, args, kwargs = task
            kwargs = kwargs.copy()
            jobs[executor.submit(fn, *args, **kwargs)] = name

    return {
        name: future.result()
        for future, name in zip(as_completed(jobs.keys()), jobs.values())
    }


def _set_context(context):
    for var, value in context.items():
        var.set(value)

Closing the issue!

[JonasKs]

Perfect😊