Spring Boot: Retry logic is never called when CircuitBreaker specifies a fallback

I’m submitting a …

• bug report
• documentation issue
• feature request

Looking for a workaround? Check Workaround below.

What is the current behaviour?

When you apply both @Retry and @CircuitBreaker annotations with fallbacks, retry logic is never called. Bellow is a slightly modified sample from the Getting Started guide to make this behaviour clearer.

@CircuitBreaker(name = BACKEND, fallbackMethod = "fallback_CB")
@Retry(name = BACKEND, fallbackMethod = "fallback_Retry")
public Mono<String> method(String param1) {
   return Mono.error(new NumberFormatException());
}

private Mono<String> fallback_Retry(String param1, RuntimeException e) {
  return Mono.just("test");
}

private Mono<String> fallback_CB(String param1, RuntimeException e) {
  return Mono.just("test");
}

Let’s create a new Spring Boot 2 application with a service like above. Then such a behavior might be observed:

• if method succeeds then neither retry nor circuit breaker logic is called. (as expected)
• if method fails then fallback_CB is called immediately, returns successfully and call finishes without calling any retry logic.
• if at some point circuit breaker get open, then after method had fails, fallback_CB is called immediately, returns successfully and call finishes without calling any retry logic. (as expected)
• if keep only @Retry(...) annotation, then on method’s constant failure, method is called for the specified amount of times and then fallback_Retry is called to finish it gracefully. (as expected)
• if you keep only @CircuitBreaker(...) annotation, then after method fails, fallback_CB is called immediately (expected or not?)
• if you keep everything as is and drop only @CircuitBreaker fallback, then retry logic works just fine and circuit breaker opens accordingly to config. But even with circuit breaker open retry makes all the “cursed to fail” calls (that might take up to a minute or so with exponential timeouts).

What is the expected behavior?

Ideally, with circuit breaker closed and a lot of calls left to make it open, logical behavior for the code above should be like that:

• When method invocation fails, retry logic is called according to configuration.
• When all retry attempts wasted, fallback_Retry is called to finish the call gracefully with fallback logic.
• If at some stage during retries circuit breaker turns open, then the next retry attempt gets intercepted and fallback_CB is called to finish it successfully (from retry perspective).
• While circuit breaker is in open state, all retry call gets immediately intercepted by circuit breaker and successfully finished by fallback_CB.

Steps to reproduce

Follow Spring Boot Get Started Guide and then check the behavior.

Note, that backing repo for the guide doesn’t use fallback specification at all.

Workaround

(for anyone searching for a solution)

After a full day of debugging and research, I was able to find a pretty nice solution that makes logic work as expected (was it intended, but not documented?).

To make it work, just change exception variable in fallback_CB to CallNotPermittedException type. Circuit breaker normally uses this exception type to notify calling code that its state is open and invocation is impossible to complete.

According to docs, fallback is called only when there is a cast from real exception to the one specified as parameter, otherwise circuit breaker throws it farther through the call stack. As in our case “Retry” is a sort of a caller, it will catch the exception and retry according to its logic. Otherwise, if circuit breaker is open, fallback_CB will be called intercepting any further retry attempts and making fallback instantaneous.

@CircuitBreaker(name = BACKEND, fallbackMethod = "fallback_CB")
@Retry(name = BACKEND, fallbackMethod = "fallback_Retry")
public Mono<String> method(String param1) {
   return Mono.error(new NumberFormatException());
}

private Mono<String> fallback_Retry(String param1, RuntimeException e) {
  return Mono.just("test");
}

private Mono<String> fallback_CB(String param1, <em>CallNotPermittedException e</em>) {
  return Mono.just("test");
}

Proposal

• Make Circuit Breaker’s fallback call fire only while open, ignoring failures in other cases (breaking change) or
• Create additional “openFallback” parameter for that specific logic or
• Fix the docs to use CallNotPermittedException in the example.

My environment:

• resilience4j: 1.17.0
• Spring Boot: 2.1.6
• Java SDK (Oracle): 12

[dlsrb6342]

@rusyasoft you can change aspect order by properties.

resilience4j:
  circuitbreaker:
    circuit-breaker-aspect-order: 100 
  retry:
    retry-aspect-order: 1000

By default, RetryAspect is higher than CircuitBreakerAspect.

[evgri243]

@RobWin, Thank you.

But in the case of ignoring exception it won’t call retry’s fallback, that doesn’t meet expectations.

As for an example or docs, it’s needed. I googled the whole internet looking for a solution or ideas, but found nothing at all. Looks like no one ever tried to do that… That’s crazy. Maybe I’m bad at googling? Docs didn’t helped me either to get that design.

Thankfully, my IDE allows to debug external libraries, so I found a way to do it by debugging…

But nevertheless, thank you for an exceptional and amazing library.