CircuitBreakerPolicy Throws on First Exception When Using Execute

Summary: I am testing out the circuit breaker policy for use in our production system and am finding that it does not wait for a consecutive number of exceptions before throwing.

I would like to log each exception and continue on. If there are 3 consecutive exceptions, then open the circuit and escalate.

Expected behavior:

Synchronous order of events:

1. Execute, throw new TimeoutException
2. Error is logged and continue.
3. Execute, throw new ArgumentNullException
4. Error is logged and continue.
5. Execute, throw new ArgumentOutOfRangeException
6. Error is logged and continue.
7. Execute, throw new Exception
8. Circuit is now open, exception is re-thrown.

Actual behaviour:

1. Execute, throw new TimeoutException
2. Circuit is now open, exception is re-thrown.

Steps / Code to reproduce the problem: Using this code in a console application should reproduce the issue at hand:

namespace PolyTestProject
{
    class Program
    {
        static void Main(string[] args)
        {
            CircuitBreakerBug();
        }
        
        static void CircuitBreakerBug()
        {
            try
            {
                var circuitBreakerPolicy = Policy
                    .Handle<TimeoutException>()
                    .CircuitBreaker(3, TimeSpan.FromSeconds(1), (exception, span) =>
                    {
                        var current = Console.ForegroundColor;
                        Console.ForegroundColor = ConsoleColor.Cyan;
                        Console.WriteLine("Why do we fall sir? So that we can learn to pick ourselves up.");
                        Console.WriteLine(exception);
                        Console.ForegroundColor = current;
                    }, () => Console.WriteLine("Resetting!"));
                
                circuitBreakerPolicy.Execute(() =>
                {
                    Console.WriteLine("Executing TimeoutException type.");
                    throw new TimeoutException("Too long!",
                        new HttpRequestException("Inner Exception!"));
                });

                circuitBreakerPolicy.Execute(() =>
                {
                    Console.WriteLine("Executing ArgumentNullException type.");
                    throw new ArgumentNullException("name",
                        new HttpRequestException("Inner Exception!"));
                });
                
                circuitBreakerPolicy.Execute(() =>
                {
                    Console.WriteLine("Executing ArgumentOutOfRangeException type.");
                    throw new ArgumentOutOfRangeException("Shouldn't be like this!",
                        new HttpRequestException("Inner Exception!"));
                });
                
                circuitBreakerPolicy.Execute(() =>
                {
                    Console.WriteLine("Executing exception type.");
                    throw new Exception("Exception!",
                        new HttpRequestException("Inner Exception!"));
                });
            }
            catch (Exception e)
            {
                var current = Console.ForegroundColor;
                Console.ForegroundColor = ConsoleColor.Red;
                Console.WriteLine(e);
                Console.ForegroundColor = current;
            }

            Console.WriteLine("Done!");
        }
    }
}

Console output:

Executing TimeoutException type.
System.TimeoutException: Too long!
 ---> System.Net.Http.HttpRequestException: Inner Exception!
   --- End of inner exception stack trace ---
   at PolyTestProject.Program.<>c.<CircuitBreakerBug>b__2_2() in /Users/matt.rhoden/Projects/PolyTestProject/PolyTestProject/Program.cs:line 76
   at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)
   at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)
   at Polly.CircuitBreaker.CircuitBreakerPolicy.<>c__DisplayClass8_0`1.<Implementation>b__0(Context ctx, CancellationToken ct)
   at Polly.CircuitBreaker.CircuitBreakerEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldHandleExceptionPredicates, ResultPredicates`1 shouldHandleResultPredicates, ICircuitController`1 breakerController)
   at Polly.CircuitBreaker.CircuitBreakerPolicy.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken)
   at Polly.Policy.Implementation(Action`2 action, Context context, CancellationToken cancellationToken)
   at Polly.Policy.Execute(Action`2 action, Context context, CancellationToken cancellationToken)
   at Polly.Policy.Execute(Action action)
   at PolyTestProject.Program.CircuitBreakerBug() in /Users/matt.rhoden/Projects/PolyTestProject/PolyTestProject/Program.cs:line 73
Done!

I might have the number of exceptions off by one, inclusive vs exclusive. However it doesn’t seem to get past the first one before raising. If I use ExecuteAndCapture, this will return a result with the exception attached to it. However it’s still not breaking after 3 times and seems like the less ideal result.

[mattrhoden]

I guess there’s no option and circuit breaker just wasn’t meant for this scenario. Thanks anyway.

[martincostello]

You would typically use a PolicyWrap to combine different policies together.