General questions about intended usage in .NET Core projects. Possible doc improvements.

Hi,

I’ve read practically all guides/issues I can find and tried many different ways of generating coverage reports with coverlet and it has left me somewhat confused. I’m a newcomer to the whole .NET universe so my questions might be trivial.

Scenario: The codebase is a multi project .NET core 3.1 solution, my intentions are to be able to generate coverage reports both local and in Azure DevOps pipelines. Both DevOps agent and local machine runs Ubuntu and Fedora respectively.

Problems

Docs suggest to avoid MSBuild variant of coverlet due to bugs. This led me to install the .NET global tool instead. All project.test.csproj files has package reference to <PackageReference Include="coverlet.collector" Version="1.0.1" />.I’ve tried invoking coverlet through a lot of commands, here are some examples:

dotnet test /p:CollectCoverage=true

dotnet test path/to/someproject.test.csproj /p:CollectCoverage=true

and

dotnet test someProject.sln --results-directory ../TestResults \
   "/p:CollectCoverage=true" \
   "/p:CoverletOutput=../TestResults/" \
   "/p:MergeWith=../TestResults/coverlet.json" \
   "/p:CoverletOutputFormat=\"json,cobertura\""

does not render any output files, errors or pretty printed summaries in the terminal like these:

Coverage summary

Output

Test Run Successful.
Total tests: 10
     Passed: 10
 Total time: 1.8588 Seconds

Calculating coverage result...
  Generating report 'pat/to/test/file/TestResults.json'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
| tested.file          | 25.33% | 18.18% | 26.38% |
+----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 25.33% | 18.18% | 26.38% |
+---------+--------+--------+--------+
| Average | 25.33% | 18.18% | 26.38% |
+---------+--------+--------+--------+

The only way I get this output is if I add the reference to ```coverlet.msbuild``` in the project.test.csproj files. Alternatively if I invoke coverlet directly on the test.dlls. However I don't get coverlet to work as a standalone command in DevOps pipelines so this is a not an viable alternative.

Questions regarding above:

• Is it correct to need coverlet.msbuild to get coverlet to work even if i invoke it by .NET global tool? Is this the best practice? Does the msbuild reference mean the test is run by msbuild instead of the global tool?
• Should it work with only coverlet.collector in the test.csprojs?
• Is it possible to give users any feedback regarding whats missing when invoking coverlet through dotnet test?

Hopefully my questions makes some kind of sense, if not, ask me and I will try to answer with more details. 😃

[Hjortsberg]

@MarcoRossignoli just a small update. We quickly struck the bugs and strange behaviour (as you explained), now running with xplat and vstest instead and it now finds all tests (so far). For anyone finding this issue, just go with Xplat/vstest straight away and save yourself some time. 😃

[MarcoRossignoli]

Docs suggest to avoid MSBuild variant of coverlet due to bugs. This led me to install the .NET global tool instead.

.NET global tool suffer of same issue as msbuild https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test .NET global tool simply instruments->run you command-> and check for coverage so it’s very similar to msbuild integration(tasks) but orchestrated by console app.

dotnet test /p:CollectCoverage=true dotnet test path/to/someproject.test.csproj /p:CollectCoverage=true

This command is the same and uses msbuild version so if you don’t add reference to coverlet.msbuild won’t work. /p:CollectCoverage=true instrument through msbuild tasks.

All project.test.csproj files has package reference to …

By design .net core test template add reference to coverlet.collector.dll because .net core platform doesn’t have a way to produce xplat. So they decided to use coverlet that is xplat. But to use coverage with collector(the preferred way due to known bugs, collectors are well used by vstest workflow) you need to pass --collect:"XPlat Code Coverage" and use “vstest integration” https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md

Is it correct to need coverlet.msbuild to get coverlet to work even if i invoke it by .NET global tool? Is this the best practice? Does the msbuild reference mean the test is run by msbuild instead of the global tool?

No if you use .net global tool(console app) the instrumentation and results will be orchestrated by console app, tool runs your command(dotnet test) after instrumenting you dll and when “your commad” exits(it’s hosted by a custom process like a normal dotnet test) the same tool will read hits file and produce coverage file.

Should it work with only coverlet.collector in the test.csprojs?

collector is needed only and only if you’re using vstest integration https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md You’ll see this reference inside every .net test template project(xunit) because is the “default” way to do coverage for every .net core app. This tool was chosed by MS and we’re proud of it.

Is it possible to give users any feedback regarding whats missing when invoking coverlet through dotnet test?

Msbuild/.NET tool integration at the moment shows on console a table with coverage percentage plus generates xml coverage file(depends on chosed format) VSTest collectors integration today doesn’t show anything because it’s pretty new and we’re working to move all feature also on this integration(that is the preferred one). For instance show to console(as you asked), merging reports and specify an absolute file path for generated file is not allowed due to vstest platform behaviour(today).