question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Is there a way to see consecutiveFailureCount property of Circuit Controller? [Expose circuit-breaker internal statistics]

See original GitHub issue

In order to intelligently arrive at a suitable count value for exceptionsAllowedBeforeBreaking for my CircuitBreaker policy, I want to be able to report on what the typical count of consecutiveFailureCount that the Circuit Breaker policy is tracking when it does ExecuteAsync. This would also be great for health reporting, to see how many times a circuit was broken on a day to day basis.

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:6 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
reisenbergercommented, Jun 1, 2019

@RichieRunner If your goal is to work out how best to configure exceptionsAllowedBeforeBreaking, you may also be interested in this discussion around circuit-breaker whatiffery. An exploration strategy can be to go initially to production with an inert circuit-breaker by setting durationOfBreak: TimeSpan.Zero. Such a circuit-breaker will still cycle through the states (you can attach logging/alerts to track when breaks occur), but with TimeSpan.Zero the breaker will have no effect on your system operation, so you can experiment with finding a suitable exceptionsAllowedBeforeBreaking value before giving the circuit breaker ‘teeth’, if you like.

1reaction
reisenbergercommented, Feb 9, 2020

Thanks @RichieRunner, that makes sense. Yes, I was focused on the second aspect of the q how many times a circuit was broken 😉

To your q: Because Polly offers two kinds of circuit-breaker (consecutive-count and advanced), implementing this is not just a simple case of add a public int ConsecutiveFailureCount to ICircuitBreakerPolicy.

We would want to tackle the general case (‘Surface circuit-breaker internal stats’) and do something like expose an ICircuitStatistics CircuitStatistics property on ICircuitBreakerPolicy.

Different implementations of ICircuitController<T> would then expose variants of ICircuitStatistics, such as IConsecutiveCountCircuitStatistics : ICircuitStatistics and IAdvancedCircuitStatistics : ICircuitStatistics. This approach would also slot into our longer-term goal of allowing users to extend circuit-breaker behaviour by implementing custom circuit controllers.

That would still leave some casting when consuming - something like (policy.CircuitStatistics as IConsecutiveCountCircuitStatistics).ConsecutiveFailureCount. To avoid that would probably mean extending the CircuitBreakerPolicy class-family with variants like ConsecutiveCountCircuitBreakerPolicy : CircuitBreakerPolicy and have the derived classes implement properties encapsulating the cast.

For clarity, priority-wise, for my own development time, this comes below distributed circuit breaker. We’d be happy to take a PR for the feature though, if you are interested to contribute one.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Implementing a Circuit Breaker with Resilience4j
A deep dive into the Resilience4j circuit breaker module. This article shows why, when and how to use it to build resilient applications....
Read more >
OPTION/PROPOSAL: Distributed Circuit Breaker · Issue #287
My first step would be to place an Azure Load Balancer (internal mode), to pass traffic to my 'set', which I would describe...
Read more >
When you use the Polly circuit-breaker, make sure ...
After some timeout, the circuit-breaker will let one method call through to "test" the API and see if it succeeds. If it fails,...
Read more >
Circuit Breaker pattern - Azure Architecture Center
The circuit breaker reverts to the Closed state after a specified number of consecutive operation invocations have been successful. If any invocation fails,...
Read more >
How to close hystrix circuit. When the circuit is "open", calls ...
How to close hystrix circuit. When the circuit is "open", calls to the circuit breaker fail immediately, without any attempt to execute the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found