Provide a way to debug a context leak

TL;DR We can store stacktrace of a RequestContext when it’s pushed via RequestContext.push() and use the information to debug a context leak.

We use the RequestContext to store and convey the information of a request. If a RequestContext is pushed into thread-local and a thread executes lines of code with the context in its thread-local, we call that the lines of code are request-scoped. Let’s see the following example:

ServiceRequestContext fooCtx = ...; // fooCtx has the information of a request from a client.
...
try (SafeCloseable ignored = fooCtx.push()) {
    // The lines in this try-with-resources
    // block are request scoped
    // until the `fooCtx` is popped by the try-with-resources.
}

Because it’s request-scoped, we can use the information of a request in the block. Let’s say that we setup to log message with the request ID using RequestContextExportingAppender:

try (SafeCloseable ignored = fooCtx.push()) {
    logger.trace("I'm tracing the flow") // This will print `RequestID=XXXX - I'm tracing the flow` to the console.
}

Because it prints the request ID, which is unique for each request, with the message, a user easily tracks the call flow of a request in an async server.

The RequestContext should be used with try-with-resources to avoid the context leak. Let’s say it’s not popped by mistake:

Executor executor = ...
executor.execute(() -> {
    SafeCloseable ignored = fooCtx.push();
    // Do some logic with fooCtx.
    ...
    // fooCtx should be popped here but it isn't
});

executor.execute(() -> {
    // This will also print `RequestID=XXXX - I am not a context of fooCtx.` because the fooCtx is not popped.
    logger.debug("I am not a context of fooCtx.")
});

In the above example, users get confused because the debug log also prints the ID of the fooCtx even though it’s irrelevant. We call that “Context leak”.

In order to prevent it, we raise an IllegalStateException when another context is pushed while the thread has a context:

Executor executor = ...
executor.execute(() -> {
    SafeCloseable ignored = fooCtx.push();
    // Do some logic with fooCtx.
    ...
    // fooCtx should be popped here but it isn't
});

executor.execute(() -> {
    ServiceRequestContext barCtx = ...
    try (SafeCloseable ignored = barCtx.push()) { // This raises an IllegalStateException.
        ...
    }
});

So we can notice that the fooCtx is not popped after its use. However, because the exception is raised when barCtx is pushed, we don’t have any stacktrace of the fooCtx. If we know when the fooCtx is pushed by looking at the stacktrace, we can easily fix the context leak.

Of course, we can scour through all lines of code to find the leak, but it’s especially hard to find it when Armeria is used with third-party or in Kotlin.

So if we can make a RequestContext has the stacktrace when it’s pushed, we can easily fix the context leak.

• RequestContext stores the stacktrace when RequestContext.push() is called.
• RequestContext removes the stored stacktrace when RequestContext.pop() is called.
• A RequestContext can be pushed multiple times before popped, we need a stack to store the stacktrace.
• When IllegalStateException is raised, it also prints the stacktrace of the current RequestContext in thread-local

Because generating stacktrace costs a lot, we can introduce different detection levels: - DISABLED: Do not record stacktrace - ADVANCED: Record only samples of RequestContexts - PARANOID: Record all RequestContexts ~~as Netty does: https://netty.io/wiki/reference-counted-objects.html#leak-detection-levels~~

We can add a flag to enable this feature or setters to ServerBuilder and ClientBuilder.

Server.builder().recordContextTrace();
// or
Server.builder().recordContextTrace(ctx -> true); // To enable recording for a specific service.

Clients.builder().recordContextTrace();

[klurpicolo]

Ok, Noted.

[minwoox]

That makes sense. 😄

@klurpicolo Let’s use the storage as @anuraaga suggested. We can create a new RequestContextStorage that tracks the stracktraces when users enable the flag here https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/internal/common/RequestContextUtil.java#L67-L115