Adding a jitter to the backoff when an aws request gets throttled.

Feature Proposal

Add exponential backoff instead of always 5 seconds.

Description

We are deploying quite often to our test AWS account. In that I noticed that we are hitting cloudformation api throttling

Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Validating template...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
...Serverless: Recoverable error occurred (Rate exceeded), sleeping for 5 seconds. Try 1 of 4
Serverless: Recoverable error occurred (Rate exceeded), sleeping for 5 seconds. Try 1 of 4
Serverless: Recoverable error occurred (Rate exceeded), sleeping for 5 seconds. Try 2 of 4
Serverless: Recoverable error occurred (Rate exceeded), sleeping for 5 seconds. Try 3 of 4
Serverless: Recoverable error occurred (Rate exceeded), sleeping for 5 seconds. Try 4 of 4

https://github.com/serverless/serverless/blob/801539ab2c1c94c5fc6d1c38c648df3fe5aec005/lib/plugins/aws/provider/awsProvider.js#L255

This is at least what I am expecting that is happending. The issue is that all my stacks will now retrying in exactly 5 seconds causing them eventually to fail.

I think there should be some kind of jitter added in order to have a higher chance of success.

[medikoo]

To be honest, I am wondering if you are trolling me 😛.

No, I want to be sure, we come with right solution, and not just take-in the code that we just assume that may improve situation.

They all potentially deploy their own stack at the same time ( at least I have no control over when they are deploying )

Adding a jitter in my understanding makes sense, when we deal with situation where AWS requests are issued at very same time (with at least 0.1s accuracy) - and that in my understanding is hard to achieve even by issuing SLS deploys automatically at very same time (there’s a lot going on in between issued AWS requests in a process, so naturally AWS requests should rather go out of sync among deploys)

Additionally above your stating that deploys potentially happen at same time. Can you elaborate on “potentially”? Are they manually released? If that’s the case, you already naturally run AWS requests with unpredictable pattern, and that should make then out of sync.

Is this then to address accidental, random cases (which may occur occasionally) where two concurrent deploys start to hit AWS SDK at about same time? As then, yes, in such scenarios, jitter can help to de-sync situation.

[philiiiiiipp]

I added support for the environment variable to change the frequency SLS_AWS_MONITORING_FREQUENCY. I was debating whether to call it Interval or Frequency, but since its already has a frequency options argument I went with this one.

I also made the calculation for the backoff simpler and provided a rounded number log.