WhatIf/dry run mode for Circuit Breakers and Bulkheads?

I’m implementing diverse policies such as Circuit Breakers and Bulkheads and having a great It Just Works experience so far - kudos to all along the way that got the code and docs to this point.

Question: A very useful thing (esp wrt tuning and bedding in within a large scale existing system) would be to be able to run them in some form of inert/monitoring/dry run mode before I give them teeth.

While this sort of whatiffery obviously doesn’t make sense in all cases, for those two it would seem it does…

(Thinking aloud: one could even run a set of fairly loose policies, and then play what-if with a tighter set of constraints and see whether and when they would have triggered or not.)

Right now, it seems the APIs don’t expose such a mechanism - i.e. my best option seems to be a mix of inspecting the state and/or catching the CircuitBrokenException, logging and then running anyway. (e.g. I don’t see an onRejecting on a circuitbreaker’s main API - though I have not spent a lot of time with the source or ILSpy yet so I could well be wrong)

I’m happy to expand on this to explain in more depth what sort of things would be of interest, but my first step is to see if I’m missing anything obvious in the APIs and/or whether anyone has had success approaching this problem in a way that’s not jumping out at me?

I’m also happy to be told ‘not our department’, in which case I’ll likely be along with questions and PRs wrt exposing some of the internal state and/or events to facilitate such inspection without Polly handling it itself.

[reisenberger]

Thanks @bartelink . Great q and can see how this would add value for test-in-prod or blue-green deployments.

Immediate options

CircuitBreaker: You can make an inert breaker right now by configuring durationOfBreak: TimeSpan.Zero. When the break threshold is met the onBreak delegate will be triggered and monitoring can be attached to that for whatiffery. No request will ever see the breaker as open, so it will be inert.

Bulkhead: You can make an inert bulkhead right now by configuring maxParallelization: Int.MaxValue. You can then monitor BulkheadAvailableCount to determine when an imaginary bulkhead capacity of N would have been exceeded.

Playing what-if with progressively tightening contraints: Dynamic reconfiguration during running (ie you can reconfigure without having to recycle the process) could support this.

Master switch: PolicyRegistry is backed by interfaces. You could easily implement a custom IReadOnlyPolicyRegistry for a master switch or “kill switch” to take Polly as a whole in/out of the mix, if desired (if that adds comfort, if experimenting in a live/prod environment). Implementation would be a custom IReadOnlyPolicyRegistry which wraps the normal implementation: ordinarily it would just delegate to a wrapped PolicyRegistry to return the normal policies, but if the “off” switch was set it could return NoOpPolicy for every Policy (taking Polly out of the mix).

---

Possible extensions to Polly

onBreakerRejected for CircuitBreaker

We would be happy to take a PR to add a delegate hook onBreakerRejected , which would operate in a similar way to onBulkheadRejected. If you/anyone want to contribute this, then I can kick-start by staking out the likely new overload signatures.

---

happy to expand on this to explain in more depth what sort of things would be of interest

Very happy to hear more …

[bartelink]

Sounds good. Best of all, I’ll hopefully be able to validate it by deleting ugly code out of CallPolly while the tests continue to pass (there’s still lots of to work to do in terms of making it more approachable, i.e. readmes etc) 😉