VSTest task should support an option to automatically rerun failed tests.

Issue Description

There are cases where functional tests or integration tests are not 100% reliable due to various reasons, including partner issues, storage account issues or some rare test race conditions issues. Investing time in fixing unreliable tests sometimes is not a short term goal, but disable them is not a good option either because those tests are still valuable in identifying real breaking changes.

Those flaky tests mostly succeeded after a retry run, it will be nice for VSTest task to enable this option to automatically rerun failed tests up to X times.

What I’m thinking how it would work:

• VSTest task runs vstest once

Continue the following retry X times

• Parse the trx result and identify failed test names
• Rerun vstest with filter set to those failed test names only(and any filters that’s specified by the user)

The question left is how to represent the test results, ideally we would want to show that individual tests failed and succeeded on those runs, but at the end signify success when all tests run succeeded after reruns.

[BrianMouncer]

@PBoraMSFT

We use YAML builds, and call VSTest.console.exe directly in some cases so that we can change test runners or run inside a docker container. I can’t figure out what “VSTest Task@2” passes through to VSTest.console.exe when using “rerunFailedTests: true”.

Do you know where this is documented, or can you just give me the information? I don’t think it is exposed on the command line, and I am thinking it is something you need to pass in via the .RunSettings file???

[TimHolzherr]

@BrianMouncer I have the exact same question 😀. Have you found an answer in the meantime?