CircuitBreaker Not Changing State from HALF_OPEN to CLOSED
See original GitHub issueI have this circuit breaker configuration in my spring-boot reactive application -
CircuitBreakerConfig.custom().failureRateThreshold(5)
.slowCallDurationThreshold(Duration.ofMillis(5000))
.slidingWindowType(SlidingWindowType.COUNT_BASED)
.slidingWindowSize(5)
.permittedNumberOfCallsInHalfOpenState(5)
.waitDurationInOpenState(Duration.ofMillis(30000))
.slowCallRateThreshold(5)
then I am calling the upstream API like this -
return Mono.from(invokeUpstream(body, headers))
.compose(CircuitBreakerOperator.of(circuitBreaker)).onErrorResume(this::fallback);
the invokeUpstream method looks like this -
Mono<ResponseEntity<String>> response = webClient.post()
.body(BodyInserters.fromValue(body))
.retrieve()
.onStatus(HttpStatus::is5xxServerError, clientResponse -> Mono.error(new BadGatewayException()))
.toEntity(String.class);
And the fallback method simply throwing exception -
private Mono<ResponseEntity<String>> fallback(Throwable ex) {
throw new BadGatewayException();
}
Now, when the upstream API returns 500 response 5 times, I can see the circuitBreaker state is moving to OPEN state from CLOSED state, which is expected. In OPEN state it stays for 30 seconds as per the configuration. After that it moves to HALF_OPEN state and the problem begins here. Even if the upstream API returns success response, it never moves to CLOSED state, it stays in HALF_OPEN state forever.
I am using these dependencies in my application -
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-circuitbreaker</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-reactor</artifactId>
<version>1.3.1</version>
</dependency>
and I am using AdoptOpenJdk11
Issue Analytics
- State:
- Created 3 years ago
- Comments:9 (3 by maintainers)
Top Results From Across the Web
CircuitBreaker Not Changing State from HALF_OPEN to ...
After that it moves to HALF_OPEN state and the problem begins here. Even if the upstream API returns success response, it never moves...
Read more >CircuitBreaker Not Changing State from HALF_OPEN to CLOSED
Hi, I am inplementing the circuit breaker over a apache http client call. evrrything works except once the circuit breaker goes in to...
Read more >Circuit Breaker - Documentation - Akka
During normal operation, a circuit breaker is in the Closed state: Exceptions or calls exceeding the configured callTimeout increment a failure counter ...
Read more >opossum 7.0.0 | Documentation - Nodeshift
Opossum is a Node.js circuit breaker that executes asynchronous functions and ... When the resetTimeout expires, opossum will enter the halfOpen state.
Read more >Circuit Breaker Statistics - Software AG Documentation
Half-Open. The first request for this service since the circuit state changes to half-open results in service execution. All other requests wait.
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Please use
CircuitBreakerUtil.isCallPermitted(cbOfY)
instead of your own if statement and invert the logic. Then configurecircuitBreakerConfig.enableAutomaticTransitionFromOpenToHalfOpen()
@RobWin I found the issue. I was checking the circuit breaker state using
circuitBreaker.tryAcquirePermission()
method even before making the actual call which was causing this issue. After removing this its working perfectly.Thank you for your support.