Stress runs

🚀 Feature Proposal

In order to ensure test quality in a repository, this feature will make sure we provide a way of stress running a test.

Stress running means to run in parallel a test, and report all the results separately, so that they can be later analized. A successful test is the one that is all green, a flaky one it’s a mix of them, and a broken test will fail in all runs.

It is important to return results separately, as people might want to later compute the flakiness percentage and decide on it (life’s not black or white! 😉).

Motivation

Ensuring that a test passes correctly before releasing it to the world helps getting higher signal when running tests, and less annoyance for other users who might be running the test later.

Example

jest --stressRuns=24 to perform 24 runs of the same test in parallel.

Pitch

Being able to run the same test at the same time requires heavy integration with the runner and the reporter. Currently you can do that by (ab)using the MPR and reporting over and over the same configuration, but that’s just a hack 😄.

[JacobMGEvans]

This still a thing? I was looking for ways to do this with the CLI and found this issue 🤣

[aaronabramov]

we should be able to specify parallelism as well (jest --stressRuns=24 -w 1) i’m currently working with a lot of e2e jest tests that spawn buck and buck takes over the entire system and loads all threads. if we run 24 bucks in parallel i’m pretty sure it’ll stall the machine 😃