Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Circuit breaker with @Async way

See original GitHub issue

Resilience4j version: 1.4.0

Java version: 11

I got one question regarding the @Async way of execution of circuit breaker and state transition.

I’m annotating my handler class with spring @Async annotation and make call of service using feign. The flow is

@Async MyHandler -> Myservice --------------> ExternalService (feign)

I’m using FeignDecorator with Circuitbreaker and retry. CircuitBreaker configuration -

            `registerHealthIndicator: true`
            `slidingWindowType: COUNT_BASED`
            `slidingWindowSize: 5`
            `minimumNumberOfCalls: 3`
            `ringBufferSizeInHalfOpenState: 1`
            `automaticTransitionFromOpenToHalfOpenEnabled: true`
            `waitDurationInOpenState: 5s #Transition to HALF_OPEN`
            `maxWaitDurationInHalfOpenState: 10s`
            `failureRateThreshold: 50`
            `eventConsumerBufferSize: 10`
            `recordExceptions:`
                - feign.FeignException
                - java.net.ConnectException
                - feign.FeignException.InternalServerError
                - feign.FeignException.FeignServerException`


Retry configuration :
       `maxRetryAttempts: 3`
       `waitDuration: 4s`   
       `retryExceptions:`
            - feign.FeignException
            - java.net.ConnectException
            - feign.FeignException.InternalServerError
            - feign.FeignException.FeignServerException

Here when MyHandler method is invoked a new thread start : executor-1 makes call to ExternalService which fails(Service is not available), it retries for 3 times and OPEN the circuit breaker. In log I see :

17-09-2020 15:15:01.854 [executor-1] DEBUG i.g.r.c.i.CircuitBreakerStateMachine.publishEventIfPossible - Event STATE_TRANSITION published: 2020-09-17T15:15:01.851616+02:00[Europe/Amsterdam]: CircuitBreaker ‘ExternalService’ changed state from CLOSED to OPEN

The handle goes to ‘CircuitBreakerAutoTransitionThread’ and transition the state to HALF_OPEN(OPEN to HALF_OPEN because I configured auto transition). ExternalService is UP now, I reprocess the record and got success response. It shows :

[CircuitBreakerAutoTransitionThread] DEBUG i.g.r.c.i.CircuitBreakerStateMachine.publishEventIfPossible - Event STATE_TRANSITION published: 2020-09-17T13:13:58.341738+02:00[Europe/Amsterdam]: CircuitBreaker ‘ExternalService’ changed state from HALF_OPEN to CLOSED

The processing is completed successfully and got CLOSED message. And then again I see this message :

[CircuitBreakerAutoTransitionThread] DEBUG i.g.r.c.i.CircuitBreakerStateMachine.publishEventIfPossible - Event STATE_TRANSITION published: 2020-09-17T15:15:21.558027+02:00[Europe/Amsterdam]: CircuitBreaker ‘ExternalService’ changed state from OPEN to HALF_OPEN

There was no failure, I think its something related to the thread executor-1

How can I fix this and make sure all state is in CLOSED? Thanks for the help!

Update : Although I see in log :

when I check the actual state of ExternalService, it shows me CLOSED :

log.info("State {} ", circuitBreakerRegistry.circuitBreaker(EXTERNALSERVICE.getValue()).getState()) -------> CLOSED

Both at the sometime like below -->

17-09-2020 17:34:29.554[CircuitBreakerAutoTransitionThread] INFO CircuitBreakerEventManager.lambda$reProcess$10 - reProcess CircuitBreaker state -->> CLOSED 17-09-2020 17:34:29.555[CircuitBreakerAutoTransitionThread] DEBUG i.g.r.c.i.CircuitBreakerStateMachine.publishEventIfPossible - Event STATE_TRANSITION published: 2020-09-17T17:34:23.938572+02:00[Europe/Amsterdam]: CircuitBreaker ‘ExternalService’ changed state from OPEN to HALF_OPEN