React to a policy state change from outside

Summary: What are you wanting to achieve? We are trying to implement something similar to what is mentioned in #287, taking into account Polly’s circuit breakers to reach a quorum on a system uptime. Our plan was to use the existing Circuit Breakers, wrapping them in a new CB policy that we could add OnBreak callbacks and react accordingly. This is because we want to let the application create their own circuit breakers and pass them in.

What code or approach do you have so far?
We have two parts, an application with a connection to a system, using a circuit breaker:

        var delay = Backoff.DecorrelatedJitterBackoffV2(settings.MedianFirstRetryDelay, settings.RetryCount);
        var retry = RetryPolicyBuilder(settings.ErrorCodes).WaitAndRetryAsync(delay, onRetry: (ex, time) => Log.Warning(ex, "Error processing request. Waiting for {RetryTimeSpan} before the next retry", time));

        var breaker = CircuitBreakerPolicyBuilder(settings.ErrorCodes).CircuitBreakerAsync(settings.ExceptionsAllowedBeforeBreaking, settings.DurationOfBreak,
            onBreak: (_, duration) => Log.Warning("Circuit breaker tripped. Circuit is open and requests won't be allowed through for {DurationOfBreak}", duration),
            onReset: () => Log.Information("Circuit closed. Requests are now allowed through"),
            onHalfOpen: () => Log.Information("Circuit is now half-opened and will test the service with the next request")
        );

        var policy = breaker.WrapAsync(retry);

        // Somewhere else
        await policy.ExecuteAsync(DoSomething)

And a library that receives a policy per system, and wraps it in a new policy updating the quorum

public void WithCircuitBreaker(Dictionary<string, ICircuitBreakerPolicy> policies)
{
    var wrapperPolicies = new Dictionary<string, ICircuitBreakerPolicy>();
    foreach (var policy in policies)
    {
        var wrapperPolicy = Policy.Handle<Exception>()
                                                .CircuitBreakerAsync(1, TimeSpan.FromMilliseconds(500), 
                                                    onBreak: IncreaseBrokenCount);
        var p = wrapperPolicy.Wrap(policy.Value);

        wrapperPolicies.Add(policy.Key, policy.Value);
    }
}

Doing this, however, the application retains the original policy, and the wrapper policy is never executed.

Is there any way of reacting to state changes in a Circuit Breaker policy, without creating a new policy wrapping it? Or to modify the onBreak callbacks to add extra behaviour to them?

Regards

[PeterCsalaHbo]

@vguzmanp Unfortunately I’m unaware of any other simpler solution 😦

[PeterCsalaHbo]

In case of V7 all onXYZ callbacks of CB are NOT allowing you to do state change. You can however manually control the CB’s state outside of the policy via the Isolate and Reset calls.

If you want to have the above described functionality then you could try the following:

• Each individual CB should register itself into a singleton component
• Individual CB’s onBreak should call into the component to indicate a failure
  • When the quorum is reached the singleton component should call the Isolate method on each CB
    • Please bear in mind the normal CB counts consecutive failures << this is pretty hard to do in the singleton

The problematic part here is the Half-Open state:

• Either you have Open and Closed states (after a predefined period of time the singleton calls Reset on each CB)
• Or you have to choose randomly a CB, call Reset on it and hope that instance will be used
  • All CBs’ onReset should call into the singleton component to reset the rest of the CBs

The solution is complex and fragile. If possible I would try to avoid to introduce this.