Improve `@RetryingTest`: offer suspend attribute or similar to configure pauses between reruns

Related to #355

User should be able to specify a “cooldown” period between failed test re-runs. Example:

@RetryingTest(value = 3, suspendForMs = 1000)
void test() {
  Assertions.fail(); // should wait 1 second (1000 ms) between runs
}

Alternatives/additions:

@RetryingTest(value = 3, suspend = 1, suspendUnit = ChronoUnit.SECONDS)
void test() {
  Assertions.fail(); // should wait 1 second (1000 ms) between runs
}

Adding other suspend modes, e.g.: RANDOM, to wait a random amount of time between 0 and suspend.

@RetryingTest(value = 3, suspend = 1000, suspendUnit = ChronoUnit.MILLIS, suspendMode = SuspendMode.RANDOM)
void test() {
  Assertions.fail(); // should wait between 0-1 second (0-1000 ms) between runs
}

Alternative:

@RetryingTest(value = 3, suspendForMs = 1000, suspendForMsFactor = 400)
void test() {
  Assertions.fail(); // should wait 1-1.4 second (1000-1400 ms) between runs
}

[slawekjaranowski]

When test depends on external resource (eg. web service) and project is developed by many developers, run on many ci systems it can happen that test from each environment will retry in the some similar time.

Of course I think about some of IT, system tests - unit test shouldn’t depends on external resources.

[nipafx]

Hey @mathieufortin01, thanks for offering to contribute. 👍🏾 You can definitely give it a go. I’m not 100% sure what would be the best way to implement this, so I recommend to start with a proof of concept (basic implementation and a few demos/tests to demonstrate) without polish (no documentation, tests don’t need to assert anything if that turns out to be complicated) and open a draft PR with that, so we can take a look.

On your two points:

• I agree, go with suspendForMs.
• While reading that I also thought about exponential backoff, shortly before you brought it up. 😆 I think that that would be a natural expansion of this concept. And while implementing randomization and backoff would probably be fairly straightforward once the basics are done, they also require documentation and tests, which would make this change much larger. Let’s go without that for the moment, so we can get a POC and eventually a merged PR reasonably soon and then expand on that in a separate PR.

PS: We’re not always timely with out replies to issues and PRs, so best prepare for that. 😊