question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CircuitBreaker: How to treat exceptions as a success

See original GitHub issue

Hi,

I have the following setup:

  1. A 3rd party service (runs on localhost:8090/sample) and it always returns bad request (400).
@RestController
@RequestMapping
public class SampleController {
    @GetMapping("/sample")
    public ResponseEntity helloWorld(HttpServletResponse response) {
        System.out.println("Request received");
        return ResponseEntity.badRequest().body("{\"message\":\"this is a bad request\"}");
    }
}
  1. In my main application where I am using circuit breaker, I am using RestTemplate to communicate it with the 3rd party service. I have ignored org.springframework.web.client.HttpClientErrorException from the circuit breaker.
  2. The circuit breaker annotation is kept at method level like below:
    @CircuitBreaker(name = "callOtherService", fallbackMethod = "errorExternal")
    public String callOtherService() {
        System.out.println("Call received");
        return restTemplate.getForObject(URI.create("http://localhost:8090/sample"), String.class);
    }

public String errorExternal(Throwable throwable) {
        return "This is error from fallback triggered by external error";
    }
  1. I am using following configuration (see callOtherService section)
resilience4j.circuitbreaker:
  configs:
    default:
      registerHealthIndicator: true
      ringBufferSizeInClosedState: 4
      ringBufferSizeInHalfOpenState: 2
      automaticTransitionFromOpenToHalfOpenEnabled: true
      waitDurationInOpenState: 20s
      failureRateThreshold: 50
      eventConsumerBufferSize: 10
      ignoreExceptions:
       - com.resilience4j.exception.BusinessException
       - feign.FeignException
      recordExceptions:
        - java.net.SocketTimeoutException
        - java.net.ConnectException
    shared:
      registerHealthIndicator: true
      ringBufferSizeInClosedState: 4
      ringBufferSizeInHalfOpenState: 2
      waitDurationInOpenState: 20s
      failureRateThreshold: 50
      eventConsumerBufferSize: 10
      ignoreExceptions:
        - com.resilience4j.exception.BusinessException
  instances:
    callOtherService:
      baseConfig: default
      ringBufferSizeInClosedState: 4
      registerHealthIndicator: true
      ignoreExceptions:
        # We need to ignore 4xx errors returned by the server as they are valid business case.
        - org.springframework.web.client.HttpClientErrorException
      recordExceptions:
        # We need to record all http errors
        - org.springframework.web.client.RestClientException

Steps to reproduce:

  1. Call the exposed endpoint 5-6 times. Verify the 3rd party service gets the call every time.
  2. Shut down the 3rd party service.
  3. Call the exposed endpoint again for 4 times. Check /actuator/health endpoint and you’ll see that the circuit is OPEN now.
  4. Wait for 20 sec.
  5. Now health endpoint says circuit is HALF_OPEN.
  6. Call exposed endpoint again for 2 times.
  7. Check health endpoint and verify circuit is OPEN again.
  8. Turn ON the 3rd party service.
  9. Verify from health endpoint if circuit is HALF_OPEN.
  10. Call the exposed endpoint. The call gets successful to 3rd party 2 times (size of ring buffer in half open state).
  11. Health endpoint still HALF_OPEN.
  12. All the calls to exposed endpoint are short-circuited and no call is attempted to the 3rd party.
  13. The circuit breaker is never CLOSED again. And there is no way to recover from this situation apart from restart the server using the circuit breaker.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:31 (24 by maintainers)

github_iconTop GitHub Comments

10reactions
RobWincommented, Aug 15, 2019

But I like the idea to add something like

recordExceptionsAsSuccess:
    - org.springframework.web.client.HttpClientErrorException
4reactions
harishmbmcommented, Aug 15, 2019

But it is not necessary for an endpoint to return 200 to show that it is running properly. If any endpoint is returning 400 (valid case if the request fails validation error) that means also that the endpoint is working, right? As per me all the 4xx errors should be treated as successful call and only 5xx should be considered as failure because the 5xx ones are the ones which are related to server health.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to Prevent Certain Exceptions from Tripping a ...
You can either ignore the exceptions you want to no longer count, or you can be explicit about the kinds of exceptions you...
Read more >
Failover and Circuit Breaker with Resilience4j | by Rob Golder
Here we see there are two overloaded fallback methods. In fact any number can be provided, with the last argument, the exception type,...
Read more >
Implementing a Circuit Breaker with Resilience4j - Reflectoring
A deep dive into the Resilience4j circuit breaker module. This article shows why, when and how to use it to build resilient applications....
Read more >
Examples - resilience4j
If you want to recover from an exception after the CircuitBreaker recorded it as a failure, you can chain the method Vavr´s. Try.recover()...
Read more >
Resilience4J Circuit Breaker to kick-in on specific HTTP status ...
By default the circuit-breaker reacts on exceptions. It records them and will open the circuit if there are too much in too less...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found