Decorrelated jitter does not jitter well

Summary: Jitter the retry delays to achieve low concurrency

I’m trying to use the suggested “decorrelated jitter” strategy

In my experiment, the suggested decorrelated jitter does not achieve what it supposed to.

First, new Random() is not quite random, when it comes to reducing concurrency. You can get multiple instances with the same seed, if many retries are issued simultaneously.

Following code achieves much lower concurrency and better distribution. In the attached screenshot you can see that the ‘advanced’ graph has much lower maximum and smoother distribution, which is highly desired to avoid concurrency spikes.

IEnumerable<TimeSpan> DecorrelatedExponent()
{
    var r = new Random(Guid.NewGuid().GetHashCode());
    for (var softAttemptCount = 0.0; softAttemptCount < 4; ) {
        softAttemptCount += r.NextDouble() * 2;
        yield return TimeSpan.FromSeconds(
            Math.Pow(2, softAttemptCount) * random.NextDouble());
    }
}

[george-polevoy]

@reisenberger Algorithm by AWS also has a flaw. On my graphs it looks smoother, but still has spikes, because it’s not truly decorrelated.

@grant-d The problem with clamping using Min Max is that it ruins randomness, flooding the distribution with some constants. As for the API guarantees, you can use math to proof the MAX and MIN. For ex. max recovery interval for retry is a sum of geometric progression, not matter if you randomize inside, if only limited range random variable does not participate in any asymptotic math function such as 1/x.

My graphing framework turned to be very useful to analyze the problem. I will share it as I promised as I clean up the code.

Currently busy preparing for my DotNext talk this week. I will follow up next week. I’ve put substantial effort into exploring this, and I will make sure we use strong math and also we will have good visual proof of what’s happening in our formulas.

[reisenberger]

@george-polevoy Are you happy if we include your ‘soft exp, soft delay X’ algorithm in the Polly.Contrib.WaitAndRetry repo? (We can do all the necessary work to move the code there.)