Feature Request: New dotnet tool command for merging results in standalone mode

N.B. Alert! 😃 I just started looking into adopting coverlet for our code coverage purpose. We run tests across many different modules not only with parallelism within the test job, but also across several different jobs running in parallel in our pipeline. Having to run tests in serial would be a huge (and unacceptable) blow back to our pipeline latency. Aside from that, I’ve noticed from several threads in here that MergeWith has its issues right now, even when executing tests in parallel. So I have a feature request, that should be fairly easy to implement. In fact, I just prototyped it for myself:

Why not have a dotnet tool command for merging existing json reports into a single final report. That way one can still do a massively parallel testing, and just run the command to merge all the results together. That command seems to be performing well even if results are merged in a single thread, but ultimately if needed, can be further enhanced to work in a multi-threaded mode using concurrent dictionaries and what not to replace the existing single-threaded data structures that are used for representing the results.

So, the pipeline would look like this (an approximation of the actual syntax just to convey the point):

- job: Build
  steps:
     - dotnet build test.sln
- job: Test
  dependsOn: Build
  strategy:
    - parallel: n
  steps:
    - dotnet test test.sln --no-build --collect:"XPlat Code Coverage" -s ./test.runsettings
    - Copy ./**/coverage.json to $(build.artifactStagingDirectory)/cc
       Flatten: true
    - publish $(build.artifactStagingDirectory)/cc
      artifact: CC_$(System.JobPositionInPhase)
- job: CodeCoverage
  dependsOn: Test
  steps:
    - download
       patterns: 'CC_*'
       path: $(pipeline.workspace)/coverageResults
    - dotnet merge-coverage-reports $(pipeline.workspace)/coverageResults/**/coverage.json -o $(pipeline.workspace)/coverageResults/merged -f json,cobertura
    - publishCodeCoverage: $(pipeline.workspace)/coverageResults/merged

Does this sound reasonable?

[jakubch1]

As part of this: https://github.com/microsoft/vstest/issues/1811 effort we will also add extensibility for other data collectors (for example coverlet). Then we also want to onboard coverlet to this logic.

At the end running dotnet test --collect "XPlat Code Coverage" should generate 1 coverage report with merged information even when running in parallel.

[MarcoRossignoli]

It has been discussed to add separate merge steps, but I can’t find the issue for that now.)

This was the discussion https://github.com/coverlet-coverage/coverlet/issues/683#issuecomment-576325784

could mean trouble going through approval process.

Report generator is used also by AzDo https://docs.microsoft.com/en-us/azure/devops/release-notes/2019/sprint-150-update#cobertura-code-coverage-report-updates FYI if you need some proof in your approval process