Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

WebClient MDC data not propagated

See original GitHub issue

Environment Details

  • Helidon Version: 2.4.2
  • Helidon SE
  • JDK version: 11
  • OS: Windows/Linux/Mac
  • Docker version (if applicable): able to reproduce in local as well as docker

Problem Description

Scenario : When Webserver Receives 1st request and does an Webclient call during Processing and Context is propagated Correctly
When Webserver Receives 2nd request and does an Webclient call during Processing , previous Webclient Context (1st one) is propagated post the Weblient call and the processing post that call is having context from previous call

Current Behavior : From 2nd Call onwards 1st call’s Context is getting propogated post webclient call Expected Behavior : Webclient call should propagate the Current Call Context without switching to first call’s context

It can be easily reprodcued Logs: 2022.03.31 15:41:56 FINE io.helidon.webserver.NettyWebServer$NettyLog Thread[nioEventLoopGroup-2-1,10,main]: [id: 0xede758a3, L:/0:0:0:0:0:0:0:0:8080] READ: [id: 0x74bb5d69, L:/0:0:0:0:0:0:0:1:8080 - R:/0:0:0:0:0:0:0:1:60235] 2022.03.31 15:41:56 FINE io.helidon.webserver.NettyWebServer$NettyLog Thread[nioEventLoopGroup-2-1,10,main]: [id: 0xede758a3, L:/0:0:0:0:0:0:0:0:8080] READ: [id: 0x27dd2624, L:/0:0:0:0:0:0:0:1:8080 - R:/0:0:0:0:0:0:0:1:60236] 2022.03.31 15:41:56 FINE io.helidon.webserver.NettyWebServer$NettyLog Thread[nioEventLoopGroup-2-1,10,main]: [id: 0xede758a3, L:/0:0:0:0:0:0:0:0:8080] READ COMPLETE 2022.03.31 15:41:56 FINE io.netty.buffer.AbstractByteBuf Thread[nioEventLoopGroup-3-2,10,main]: -Dio.netty.buffer.checkAccessible: true 2022.03.31 15:41:56 FINE io.netty.buffer.AbstractByteBuf Thread[nioEventLoopGroup-3-2,10,main]: -Dio.netty.buffer.checkBounds: true 2022.03.31 15:41:56 FINE io.netty.util.ResourceLeakDetectorFactory Thread[nioEventLoopGroup-3-2,10,main]: Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@4f79c007 2022.03.31 15:41:56 FINE io.netty.util.Recycler Thread[nioEventLoopGroup-3-2,10,main]: -Dio.netty.recycler.maxCapacityPerThread: 4096 2022.03.31 15:41:56 FINE io.netty.util.Recycler Thread[nioEventLoopGroup-3-2,10,main]: -Dio.netty.recycler.ratio: 8 2022.03.31 15:41:56 FINE io.netty.util.Recycler Thread[nioEventLoopGroup-3-2,10,main]: -Dio.netty.recycler.chunkSize: 32 2022.03.31 15:41:56 FINE io.netty.util.Recycler Thread[nioEventLoopGroup-3-2,10,main]: -Dio.netty.recycler.blocking: false 2022.03.31 15:41:56 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-1,10,main]: [Handler: 1746738214, Channel: 0x74bb5d69] Read complete not lastContent 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received HttpRequest: 1027168240. Remote address: /0:0:0:0:0:0:0:1:60236. Scope id: web-1:1 Setting Namespace : 01 MDC_NAMESPACE_KEY After set: Optional[01] 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received HttpContent: 1673502283 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received HttpContent: 1110988450 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received LastHttpContent: 1110988450 MDC_NAMESPACE_KEY: Optional[01] 2022.03.31 15:41:57 INFO org.example.GreetService Thread[nioEventLoopGroup-3-2,10,main]: Initial Namespace in processMessageHandler map :01 MDC_NAMESPACE_KEY: Optional[01] 2022.03.31 15:41:57 INFO org.example.GreetService Thread[nioEventLoopGroup-3-2,10,main]: Namespace in webclientCall :01 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Read complete lastContent 2022.03.31 15:41:57 FINE io.helidon.webserver.NettyWebServer$NettyLog Thread[nioEventLoopGroup-2-1,10,main]: [id: 0xede758a3, L:/0:0:0:0:0:0:0:0:8080] READ: [id: 0x3faa0467, L:/127.0.0.1:8080 - R:/127.0.0.1:60239] 2022.03.31 15:41:57 FINE io.helidon.webserver.NettyWebServer$NettyLog Thread[nioEventLoopGroup-2-1,10,main]: [id: 0xede758a3, L:/0:0:0:0:0:0:0:0:8080] READ COMPLETE 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Received HttpRequest: 1299183979. Remote address: /127.0.0.1:60239. Scope id: web-1:2 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Read complete has requests: io.helidon.webserver.RequestContext@66409399 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Received HttpContent: 1932320011 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Received HttpContent: 93927352 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Received LastHttpContent: 93927352 2022.03.31 15:41:57 FINE io.netty.handler.codec.compression.Brotli Thread[helidon-client-0,10,main]: brotli4j not in the classpath; Brotli support will be unavailable. 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Response complete: 1299183979 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Read complete lastContent MDC_NAMESPACE_KEY: Optional[01] 2022.03.31 15:41:57 INFO org.example.GreetService Thread[helidon-client-0,10,main]: Namespace post Webclient Call in processMessageHandler accept :01 2022.03.31 15:41:57 INFO org.example.GreetService Thread[helidon-client-0,10,main]: { "status": "Success" } 2022.03.31 15:41:57 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Response complete: 1027168240 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received HttpRequest: 77709676. Remote address: /0:0:0:0:0:0:0:1:60236. Scope id: web-1:3 Setting Namespace : 02 MDC_NAMESPACE_KEY After set: Optional[02] 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received HttpContent: 1725809056 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Read complete has requests: io.helidon.webserver.RequestContext@e2e56ae 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received HttpContent: 68045990 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Received LastHttpContent: 68045990 MDC_NAMESPACE_KEY: Optional[02] 2022.03.31 15:42:09 INFO org.example.GreetService Thread[nioEventLoopGroup-3-2,10,main]: Initial Namespace in processMessageHandler map :02 MDC_NAMESPACE_KEY: Optional[02] 2022.03.31 15:42:09 INFO org.example.GreetService Thread[nioEventLoopGroup-3-2,10,main]: Namespace in webclientCall :02 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Read complete lastContent 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Received HttpRequest: 1486904527. Remote address: /127.0.0.1:60239. Scope id: web-1:4 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Read complete has requests: io.helidon.webserver.RequestContext@57698351 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Received HttpContent: 1591854374 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Received LastHttpContent: 1591854374 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Response complete: 1486904527 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-3,10,main]: [Handler: 141393570, Channel: 0x3faa0467] Read complete lastContent MDC_NAMESPACE_KEY: Optional[01] 2022.03.31 15:42:09 INFO org.example.GreetService Thread[helidon-client-0,10,main]: Namespace post Webclient Call in processMessageHandler accept :01 2022.03.31 15:42:09 INFO org.example.GreetService Thread[helidon-client-0,10,main]: { "status": "Success" } 2022.03.31 15:42:09 FINE io.helidon.webserver.ForwardingHandler Thread[nioEventLoopGroup-3-2,10,main]: [Handler: 1110999505, Channel: 0x27dd2624] Response complete: 77709676

Steps to reproduce

Sample Code Link : https://github.com/vamseeds/HelidonContextPropagation Steps to reproduce : Call 1 : http://localhost:8080/01/message (Can see the current Context logs) Call 2 : http://localhost:8080/02/message (Will start seeing Call 1 logs post Webclient Call)

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
romain-grecourtcommented, Apr 4, 2022

A workaround for this is to manually propagate the MDC data using context. E.g. In your filter do something like this:

 String ws = req.path().param("ws");
 HelidonMdc.set("namespace-key", ws);
 req.context().register("namespace-key", ws);
 req.next();

In the webclient response callback:

HelidonMdc.set("namespace-key", req.context()
                                   .get("namespace-key", String.class)
                                   .orElse("null"));
1reaction
romain-grecourtcommented, Mar 31, 2022

I was able to fix this in master with the following changes:

diff --git a/webclient/webclient/src/main/java/io/helidon/webclient/WebClientRequestBuilderImpl.java b/webclient/webclient/src/main/java/io/helidon/webclient/WebClientRequestBuilderImpl.java
index 8c3ebd0ce..39a209c18 100644
--- a/webclient/webclient/src/main/java/io/helidon/webclient/WebClientRequestBuilderImpl.java
+++ b/webclient/webclient/src/main/java/io/helidon/webclient/WebClientRequestBuilderImpl.java
@@ -29,6 +29,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
+import java.util.ServiceLoader;
 import java.util.Set;
 import java.util.StringTokenizer;
 import java.util.concurrent.CompletableFuture;
@@ -45,6 +46,7 @@ import java.util.logging.Logger;
 import io.helidon.common.GenericType;
 import io.helidon.common.context.Context;
 import io.helidon.common.context.Contexts;
+import io.helidon.common.context.spi.DataPropagationProvider;
 import io.helidon.common.http.DataChunk;
 import io.helidon.common.http.Headers;
 import io.helidon.common.http.Http;
@@ -52,6 +54,7 @@ import io.helidon.common.http.HttpRequest;
 import io.helidon.common.http.MediaType;
 import io.helidon.common.http.Parameters;
 import io.helidon.common.reactive.Single;
+import io.helidon.common.serviceloader.HelidonServiceLoader;
 import io.helidon.media.common.MessageBodyReadableContent;
 import io.helidon.media.common.MessageBodyReaderContext;
 import io.helidon.media.common.MessageBodyWriterContext;
@@ -83,6 +86,9 @@ class WebClientRequestBuilderImpl implements WebClientRequestBuilder {
     private static final Logger LOGGER = Logger.getLogger(WebClientRequestBuilderImpl.class.getName());
 
     private static final Map<ConnectionIdent, Set<ChannelRecord>> CHANNEL_CACHE = new ConcurrentHashMap<>();
+    private static final List<DataPropagationProvider> PROPAGATION_PROVIDERS = HelidonServiceLoader
+            .builder(ServiceLoader.load(DataPropagationProvider.class)).build().asList();
+
     static final AttributeKey<WebClientRequestImpl> REQUEST = AttributeKey.valueOf("request");
     static final AttributeKey<CompletableFuture<WebClientServiceResponse>> RECEIVED = AttributeKey.valueOf("received");
     static final AttributeKey<CompletableFuture<WebClientServiceResponse>> COMPLETED = AttributeKey.valueOf("completed");
@@ -612,10 +618,15 @@ class WebClientRequestBuilderImpl implements WebClientRequestBuilder {
             });
             return result;
         }));
-
         return wrapWithContext(single);
     }
 
+    @SuppressWarnings(value = "unchecked")
+    private void runInContext(Map<Class<?>, Object> data, Runnable command) {
+        PROPAGATION_PROVIDERS.forEach(provider -> provider.propagateData(data.get(provider.getClass())));
+        Contexts.runInContext(context, command);
+    }
+
     /**
      * Wraps a single into another that runs all subscriber methods using the current
      * context. This will enable calls to {@code Contexts.context()} in reactive handlers
@@ -626,25 +637,27 @@ class WebClientRequestBuilderImpl implements WebClientRequestBuilder {
      * @return wrapped single
      */
     private <T> Single<T> wrapWithContext(Single<T> single) {
+        Map<Class<?>, Object> contextProperties = new HashMap<>();
+        PROPAGATION_PROVIDERS.forEach(provider -> contextProperties.put(provider.getClass(), provider.data()));
         return Single.create(subscriber -> single.subscribe(new Flow.Subscriber<T>() {
             @Override
             public void onSubscribe(Flow.Subscription subscription) {
-                Contexts.runInContext(context, () -> subscriber.onSubscribe(subscription));
+                runInContext(contextProperties, () -> subscriber.onSubscribe(subscription));
             }
 
             @Override
             public void onNext(T item) {
-                Contexts.runInContext(context, () -> subscriber.onNext(item));
+                runInContext(contextProperties, () -> subscriber.onNext(item));
             }
 
             @Override
             public void onError(Throwable throwable) {
-                Contexts.runInContext(context, () -> subscriber.onError(throwable));
+                runInContext(contextProperties, () -> subscriber.onError(throwable));
             }
 
             @Override
             public void onComplete() {
-                Contexts.runInContext(context, subscriber::onComplete);
+                runInContext(contextProperties, subscriber::onComplete);
             }
         }));
     }
Read more comments on GitHub >

github_iconTop Results From Across the Web

WebClient MDC data not propagated · Issue #4089 - GitHub
A workaround for this is to manually propagate the MDC data using context. E.g. In your filter do something like this: String ws...
Read more >
How does Spring Cloud Sleuth propagate the MDC context in ...
The request started in a reactor-http-nio thread , and then swiched to boundedElastic-2. As you can see the traceId and spanId values are...
Read more >
MDC with WebClient in WebMVC - Tadaya's Blog
We can use Schedulers. addExecutorServiceDecorator to return wrapped ScheduledExecutorService that propagates MDC values to the new thread. ...
Read more >
Project Reactor + MDC logging - Medium
This post provides technique for propagating MDC context over reactive pipeline of Mono/Flux handlers with small impact on performance and ...
Read more >
Passing Context with Spring WebFlux (Part I) - Nicolas Portmann
Such contextual data might include but is not limited to the following: ... public class MdcHeaderFilter implements WebFilter { @Override
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Bug: ConfigProperty annotation on Map<String, String> field doesn't filter the configuration values, but populates all configuration entries in Map

Next page

HelidonSE - DataChunkInputStream never reads bytes