question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CircuitBreaker Not Changing State from HALF_OPEN to CLOSED

See original GitHub issue

I have this circuit breaker configuration in my spring-boot reactive application -

CircuitBreakerConfig.custom().failureRateThreshold(5)
            .slowCallDurationThreshold(Duration.ofMillis(5000))
            .slidingWindowType(SlidingWindowType.COUNT_BASED)
            .slidingWindowSize(5)
            .permittedNumberOfCallsInHalfOpenState(5)
            .waitDurationInOpenState(Duration.ofMillis(30000))
            .slowCallRateThreshold(5)

then I am calling the upstream API like this -

return Mono.from(invokeUpstream(body, headers))
                .compose(CircuitBreakerOperator.of(circuitBreaker)).onErrorResume(this::fallback);

the invokeUpstream method looks like this -

Mono<ResponseEntity<String>> response = webClient.post()
                                                   .body(BodyInserters.fromValue(body))
                                                   .retrieve()
                                                   .onStatus(HttpStatus::is5xxServerError, clientResponse -> Mono.error(new BadGatewayException()))
                                                   .toEntity(String.class);

And the fallback method simply throwing exception -

private Mono<ResponseEntity<String>> fallback(Throwable ex) {
    throw new BadGatewayException();
}

Now, when the upstream API returns 500 response 5 times, I can see the circuitBreaker state is moving to OPEN state from CLOSED state, which is expected. In OPEN state it stays for 30 seconds as per the configuration. After that it moves to HALF_OPEN state and the problem begins here. Even if the upstream API returns success response, it never moves to CLOSED state, it stays in HALF_OPEN state forever.

I am using these dependencies in my application -

<dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-circuitbreaker</artifactId>
        <version>1.3.1</version>
</dependency>
<dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-reactor</artifactId>
        <version>1.3.1</version>
</dependency>

and I am using AdoptOpenJdk11

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
RobWincommented, May 18, 2020

Please use CircuitBreakerUtil.isCallPermitted(cbOfY) instead of your own if statement and invert the logic. Then configure circuitBreakerConfig.enableAutomaticTransitionFromOpenToHalfOpen()

2reactions
bivrantoshakilcommented, May 15, 2020

@RobWin I found the issue. I was checking the circuit breaker state using circuitBreaker.tryAcquirePermission() method even before making the actual call which was causing this issue. After removing this its working perfectly.

Thank you for your support.

Read more comments on GitHub >

github_iconTop Results From Across the Web

CircuitBreaker Not Changing State from HALF_OPEN to ...
After that it moves to HALF_OPEN state and the problem begins here. Even if the upstream API returns success response, it never moves...
Read more >
CircuitBreaker Not Changing State from HALF_OPEN to CLOSED
Hi, I am inplementing the circuit breaker over a apache http client call. evrrything works except once the circuit breaker goes in to...
Read more >
Circuit Breaker - Documentation - Akka
During normal operation, a circuit breaker is in the Closed state: Exceptions or calls exceeding the configured callTimeout increment a failure counter ...
Read more >
opossum 7.0.0 | Documentation - Nodeshift
Opossum is a Node.js circuit breaker that executes asynchronous functions and ... When the resetTimeout expires, opossum will enter the halfOpen state.
Read more >
Circuit Breaker Statistics - Software AG Documentation
Half-Open. The first request for this service since the circuit state changes to half-open results in service execution. All other requests wait.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found