Coverage results aren't stable

We (dotnet/runtime) recently switched from coverlet’s global tool to dotnet test with the coverlet data collector. With the data collector driver I just noticed that the produce coverage results file isn’t stable and varies between multiple invocations.

See the generated reports for five invocations of C:\git\runtime\src\libraries\System.Text.RegularExpressions\tests>dotnet test -f netcoreapp5.0 /p:Coverage=true: coverage-RegularExpressions.zip

Something that immediately points out is the difference in the summary:

  <Summary numSequencePoints="10557" visitedSequencePoints="7413" numBranchPoints="4982" visitedBranchPoints="3491" sequenceCoverage="70.21" branchCoverage="70.07" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="53" numClasses="55" visitedMethods="727" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="7431" numBranchPoints="4982" visitedBranchPoints="3516" sequenceCoverage="70.38" branchCoverage="70.57" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="53" numClasses="55" visitedMethods="727" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="8415" numBranchPoints="4982" visitedBranchPoints="4107" sequenceCoverage="79.71" branchCoverage="82.43" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="54" numClasses="55" visitedMethods="768" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="8181" numBranchPoints="4982" visitedBranchPoints="4029" sequenceCoverage="77.49" branchCoverage="80.87" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="54" numClasses="55" visitedMethods="758" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="8376" numBranchPoints="4982" visitedBranchPoints="4092" sequenceCoverage="79.34" branchCoverage="82.13" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="54" numClasses="55" visitedMethods="768" numMethods="817" />

VSTest reports the same number of tests being discovered and executed for all the five runs:

Test Run Successful.
Total tests: 7204
     Passed: 7204

I’m wondering if hit points aren’t correctly propagated back to the hits file?

cc @MarcoRossignoli @sharwell @petli @stephentoub

[ViktorHofer]

I didn’t roll anything back 😃 You can try it in dotnet/runtime master with:

build.cmd clr+libs -rc Release
dotnet test -f net5.0 /p:Coverage=true src\libraries\System.Text.RegularExpressions\tests\

[ViktorHofer]

So the instrumentation is working as expected as the numSequencePoints, numBranchPoints, maxCyclomaticComplexity, minCyclomaticComplexity, numClasses and numMethods values are the same.

The visited ones differ largely…