VsTest Code Coverage is broken by design

Environment

• Azure DevOps Server 2019 Update 1 Patch 1 (Dev17.M153.3)
• Agent v2.153.2 inside Docker Container based on mcr.microsoft.com/dotnet/framework/runtime:4.8-20190709-windowsservercore-1903
• Microsoft.TestPlatform v16.3.0 installed via VsTestPlatformToolInstaller

Issue Description

With codeCoverageEnabled: true the VsTest task generates a proprietary .coverage file which cannot be consumed by Azure DevOps / PublishCodeCoverageResults nor is it published as an artifact. The only supported formats are Cobertura and JaCoCo. Reading the docs for PublishCodeCoverageResults it says :

Tasks such as Visual Studio Test, .NET Core, Ant, Maven, Gulp, Grunt also provide the option to publish code coverage data to the pipeline. If you are using these tasks, you do not need a separate Publish Code Coverage Results task in the pipeline.

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results

This is wrong because the VsTest task neither converts the resulting .coverage to a supported file format nor uploads the raw .coverage file as an artifact.

But the story continues…

So if the task itself does not produce a valid output why not simply convert the .coverage file to Cobertura using danielpalme/ReportGenerator and then use PublishCodeCoverageResults ?

Unfortunately this is also not possible because of #6279. The .coverage file does not exist anymore after a test run so there is nothing we could do with it. This effectively renders the complete code coverage feature of the VsTest task useless.

Obviously we could use a simple script to execute our tests with enabled code coverage and then do the file conversion, but then we would also loos all the nice features of the VsTest task like distributed batching and flaky test detection.

Proposal

It would be great to get some generic infrastructure to use any code coverage tool which would resolve the issue in the first place. Because most of the the time these tools works the same (you just run them with an argument telling them what other executable to run) this should be quite simple?

Degenerated example using dotCover: dotCover.exe cover --TargetExecutable=$(vsTestExecutable) --TargetArguments=$(vsTestArguments) --ReturnTargetExitCode

[p00j4]

This issue needs to be Re - Opened because the proposed feature enable to display the html by default never came to light. And every new user of VsTest task integrating on the ADO is facing the same race condition.

• Can’t move to powershell to manage whole test lifecycle by self
• And can’t use VsTest task because the .coverage is not displayable.

Proposal: why can’t codecoverage (from VsTest task) also output the .xml by default? It’s upto the use to consume it or not.

[simonachmueller]

@ShreyasRmsft @anshii03 this issue needs to be definitely re-opened, we are facing a similar problem (very well described here https://github.com/microsoft/vstest/issues/981#issuecomment-1034093540)