Add exponential backoff to the retry algorithm of WebCmdlets

Summary of the new feature / enhancement

I propose adding an optional parameter to Invoke-WebRequest and Invoke-RestMethod to determine the retry interval based on the exponential backoff algorithm.

This feature was discussed back when a proposal to implement basic retry functionality in WebCmdlets in #5582. However, at that time, it was decided to focus only on implementing very basic retry function and postpone this feature. I would like to reopen this feature proposal.

This is a method that increases the retry interval exponentially, such as 1 second, 2 seconds, 4 seconds, 8 seconds. It is a recommended retry method when making requests to API endpoints of many cloud services such as Azure and AWS.

curl uses this algorithm by default. It is also implemented in Azure SDK and AWS SDK.

To avoid breaking changes, the default mechanism should not be changed, and it should be selectable by an optional parameter. There may be room for discussion about the name of the parameter, but how about -RetryMode?

-RetryMode [string] <Fixed | Exponential>

When using it, combine with the existing parameters -MaximumRetryCount and -RetryIntervalSec.

Example 1:

Invoke-WebRequest -Uri $Uri -MaximumRetryCount 5 -RetryIntervalSec 1 -RetryMode Exponential

This code will make a maximum of 6 requests, including the first request, with the following intervals: Request intervals: 0s > 1s > 2s > 4s > 8s > 16s

Example 2:

If -RetryMode is speccified as Fixed or not speccified the parameter, retrying will be at a fixed interval. This is the current behavior of WebCmdlets.

Invoke-WebRequest -Uri $Uri -MaximumRetryCount 5 -RetryIntervalSec 2 -RetryMode Fixed

Request intervals: 0s > 2s > 2s > 2s > 2s > 2s

Example 3:

If -RetryMode is speccified as Exponential and not speccified the -RetryIntervalSec, it will start from the default value of 5 seconds.

Invoke-WebRequest -Uri $Uri -MaximumRetryCount 5 -RetryMode Exponential

Request intervals: 0s > 5s > 10s > 20s > 40s > 80s

Jitter

It may also be worth considering adding a feature to apply jitter to the retry interval. This prevents retries from colliding when multiple requests are executed simultaneously. The exponential backoff algorithm with jitter is often the best retry algorithm for public endpoints.

Considering the consistency with other parameters, how about -MaximumRetryJitterSec for the parameter name?

Example:

Invoke-WebRequest -Uri $Uri -MaximumRetryCount 5 -RetryIntervalSec 1 -RetryMode Exponential -MaximumRetryJitterSec 1

It adds a random jitter between 0 and 1 second to the retry interval. The request intervals will be as follows: Request intervals: 0s > 1.8s > 2.3s > 4.0s > 8.6s > 16.3s

Note:

Regardless of what values are specified for -RetryMode or -RetryIntervalSec, if the HTTP status code is 429 and there is the Retry-After in the response header, always respect that value.

[mkht]

@CarloToso Thanks for agreeing to my proposal. I understand that… 😃

For the moment I will work on the implementation and submit a PR, but I would like to hear a bit more from others about the parameter names and max interval values.

[iSazonov]

@mkht I have no influence on WGs. I guess they don’t even have any regulation and responsibility for the commitments they make. I can only advise you to directly ping the members of the group that is responsible for this area.

---

@SteveL-MSFT What do you think about replacing inactive workgroup members? I think some groups don’t work at all. Maybe make their work more public? For example, use GitHub Projects so that we can see which issues are at what stage of consideration and what the results of their work are. Or they could have used more stage labels.