Sleuth - Not setting MDC with new span id when request is sent with WebClient

WebClient Initialisation

Code to call backend and initialisation as shown below. The sample code is using the spring-cloud-version Hoxton.SR4 , ‘org.springframework.boot’ version ‘2.3.1.RELEASE’. Full sample at https://github.com/malika15/sleuth-resttemplate-webclient-comparison

@EnableAutoConfiguration
@RestController
public class FrontendUsingWebClient {

    @Autowired
    WebClient webClient;
    private Logger log = LoggerFactory.getLogger("FrontendUsingWebClient");

    public static void main(String[] args) {
        SpringApplication.run(FrontendUsingWebClient.class,
            "--spring.application.name=FrontendUsingWebClient",
            "--server.port=8081");
    }

    @Bean
    WebClient webClient() {
        return WebClient.builder().exchangeStrategies(ExchangeStrategies.builder().codecs(c ->
            c.defaultCodecs().enableLoggingRequestDetails(true)).build()
        ).baseUrl("http://localhost:9000").build();
    }

    @RequestMapping("/")
    public Mono<String> callBackend() {
        log.info("Frontend WebClient::Begin");
        Mono<String> response = webClient.get().uri("/api").retrieve().bodyToMono(String.class)
            .doFirst(() -> log.info("Frontend WebClient ::doFirst"))
            .doOnSubscribe(subscription -> log.info("Frontend WebClient::doOnSubscribe"))
            .doOnRequest(value -> log.info("Frontend WebClient::doOnRequest"))
            .doOnSuccess(s -> log.info("Frontend WebClient::doOnSuccess"))
            .doOnEach(stringSignal -> log.info("Frontend WebClient::doOnEach"))
            .doOnNext(s -> log.info("Frontend WebClient::doOnNext"))
            .doAfterTerminate(() -> log.info("Frontend::doAfterTerminate"))
            .doFinally(signalType -> log.info("Frontend WebClient::doFinally"))
            .doOnCancel(() -> log.info("Frontend WebClient::doOnCancel"));
        log.info("Frontend WebClient::End");
        return response;
    }
}

Webclient - New span id in HTTP headers but not in MDC

Notice in log output below, HTTP header has X-B3-SpanId:"e8c842b87aa2c176" where as MDC has d6737613f8c125f7,d6737613f8c125f7. Sleuth starts a new span for the webclient call and sets it on request X-B3 Headers but fails to set on the TraceContext(MDC)

2020-06-19 13:55:29.516 TRACE [FrontendUsingWebClient,d6737613f8c125f7,d6737613f8c125f7,false] 14008 --- [ctor-http-nio-3] o.s.w.r.f.client.ExchangeFunctions       : [418bc045] HTTP GET http://localhost:9000/api, headers=[X-B3-TraceId:"d6737613f8c125f7", X-B3-SpanId:"e8c842b87aa2c176", X-B3-ParentSpanId:"d6737613f8c125f7", X-B3-Sampled:"0"]

RestTemplate - New span id in HTTP headers and in MDC

Notice in log output below new span being created Starting scope for span: 2d795955f8643a11/79ec6c794e86cd58 and set on MDC 2d795955f8643a11,79ec6c794e86cd58. Sleuth starts new span for RestTemplate call and closes after response is received. Setting it on MDC appropriately

2020-06-19 13:52:15.520 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.web.client.RestTemplate              : HTTP GET http://localhost:9000/api
2020-06-19 13:52:15.524 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.web.client.RestTemplate              : Accept=[text/plain, application/json, application/*+json, */*]
2020-06-19 13:52:15.525 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.b.f.s.DefaultListableBeanFactory     : Returning cached instance of singleton bean 'tracingClientHttpRequestInterceptor'
2020-06-19 13:52:15.526 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] o.s.c.sleuth.log.Slf4jScopeDecorator     : Starting scope for span: 2d795955f8643a11/79ec6c794e86cd58
2020-06-19 13:52:15.526 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] o.s.c.sleuth.log.Slf4jScopeDecorator     : With parent: 3276748429663156753
2020-06-19 13:52:15.526  INFO [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] FrontendUsingRestTemplate                : Frontend RestTemplate::Sending request to Backend http://localhost:9000/api
2020-06-19 13:52:15.533 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection  : ProxySelector Request for http://localhost:9000/api
2020-06-19 13:52:15.535 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection  : Proxy used: DIRECT
2020-06-19 13:52:15.536 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection  : sun.net.www.MessageHeader@161711d99 pairs: {GET /api HTTP/1.1: null}{Accept: text/plain, application/json, application/*+json, */*}{X-B3-TraceId: 2d795955f8643a11}{X-B3-SpanId: 79ec6c794e86cd58}{X-B3-ParentSpanId: 2d795955f8643a11}{X-B3-Sampled: 0}{User-Agent: Java/11.0.2}{Host: localhost:9000}{Connection: keep-alive}
2020-06-19 13:52:15.766 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection  : KeepAlive stream used: http://localhost:9000/api
2020-06-19 13:52:15.768 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection  : sun.net.www.MessageHeader@1c1638c03 pairs: {null: HTTP/1.1 200 OK}{Content-Type: text/plain;charset=UTF-8}{Content-Length: 39}
2020-06-19 13:52:15.769 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] o.s.c.sleuth.log.Slf4jScopeDecorator     : Closing scope for span: 2d795955f8643a11/79ec6c794e86cd58
2020-06-19 13:52:15.771 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.web.client.RestTemplate              : Response 200 OK

Looking for ideas on how to get span generated for Webclient calls into MDC for it to show up in the logs for request and response

[ikarsokolov]

It worth mentioning that sample by @malika15 does not work with Spring Boot 2.2.8.RELEASE as well.

[marcingrzejszczak]

Webclient instrumentation works fine. It’s integration with reactor that is problematic. Currently there’s no other way to make it better in a transparent way however if you have an idea how to fix this then please be my guest and file a pr with a change. We’ll be more than happy to review it