question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Propagate distributed tracing information across Thread pool bulkhead

See original GitHub issue

Using using Spring Cloud Sleuth distributed tracing, one needs to take care to handover trace information across threads.

The way Sleuth does this is generally by wrapping beans in the application context with tracing ones. Specifically in the case TraceableExecutorService.

Currently in resilience4j-spring-boot2 I see no way in ThreadPoolBulkheadConfiguration to provide an ExexcutorService factory or decorator

Resilience4j version: 1.1.0

Java version: 11

Problem description:

Spring Cloud Sleuth propagates tracing by default using ThreadLocal. When handing over tasks to a thread pool care must be taken to create a new Span based on the current Span such that this action is recorded in the trace.

The usual approach to this is to [decorate the Runnable or Callable] with a tracing aware option that grabs the current context on construction and wraps execution in a scoped Span.

The Resilience4j Spring Boot 2 integration using @Bulkhead wraps method calls in an aspect, which submits the execution to an ExecutorService in FixedThreadPoolBulkhead. This is bootstrapped through ThreadPoolBulkheadConfiguration. As the ExecutorService is created in the bulkhead’s construction there is no way to wrap the ExecutorService in a trace aware one. There is also no mechanism to supply a custom Callable wrapper to BulkheadAspect#proceedInThreadPoolBulkhead.

This precludes the use of a thread pool based bulkhead in a Spring Sleuth environment.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:1
  • Comments:27 (13 by maintainers)

github_iconTop GitHub Comments

2reactions
markward-schubertcommented, Jun 9, 2021

Thanks for the feedback @RobWin !

My main problem is, that I do not have a clue how to properly implement a SpringSleuthTraceContextPropagator 😃, so how to do the concrete implementation. I hoped you guys might have done this already for some future integration with spring-cloud.

2reactions
bmorris591commented, Nov 13, 2019

Thanks for the response - I saw that discussion.

But I think my feature request is very different - this is about customisations of objects created as part of the Spring Boot integrations in an idiomatic way.

That thread is solely about passing thread local.

This request is not to change the Resilience4j API (although subtle changes may be required - like IoC with the thread pool) but to change the Spring Boot starter to have the ability to modify the created objects.

Happy to have the solution discussed in the other thread - but that only answers part of the question. This enhancement to the Spring Boot integration would still be needed to allow access to the enhancement discussed in the other issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Implementing Bulkhead with Resilience4j - Reflectoring
The ThreadPoolBulkhead uses a thread from a thread pool to execute our code. It internally uses a java.util.concurrent.
Read more >
What is Context Propagation in Distributed Tracing? - SigNoz
Distributed tracing is built on causal metadata context propagation. It aims to capture events in a sequential flow that depicts the causal ...
Read more >
Bulkhead - resilience4j
The SemaphoreBulkhead should work well across a variety of threading and I/O models. It is based on a semaphore, and unlike Hystrix, does...
Read more >
WildFly Swarm Microservices on Red Hat OpenShift Container ...
While the Jaeger service allows distributed tracing data to be aggregated, persisted and used for reporting, this application also relies on the client-side ......
Read more >
Context propagation in distributed tracing: Beyond "Hello ...
Context propagation in distributed tracing: Beyond “Hello World” examples · Unlocking the magic of distributed tracing across all kinds of ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found