Coverlet producing incomplete coverage results

Hi there, we have kind of an oddball situation here which is potentially causing issues with Coverlet. This is a client project / client code so unfortunately I can only share the set-up and not the actual bits and pieces.

The solution is quite large with a total of 128 projects. These are all .NET Framework 4.8 SDK-style. We have WebAPI projects in there which normally you cannot use with SDK style, but using https://github.com/CZEMacLeod/MSBuild.SDK.SystemWeb we have converted these to SDK style as well. We’re not using any test frameworks like xunit or such, and all package versions are on the latest (MSTest / Test SDK and Coverlet as well).

Due to the SDK style projects we cannot use dotnet test, that just does not work. We can use dotnet vstest however, or simply call vstest.console.exe so that is what we do. Coverlet collectors is set-up using a runsettings file which is not that special:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
	<RunConfiguration>
		<TreatNoTestsAsError>false</TreatNoTestsAsError>
		<TargetPlatform>x64</TargetPlatform>
	  <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
	</RunConfiguration>
	<LoggerRunSettings>
		<Loggers>
			<Logger friendlyName="trx" enabled="True"/>
		</Loggers>
	</LoggerRunSettings>
       <!-- Configurations for data collectors -->
	<DataCollectionRunSettings>
		<DataCollectors>
			<DataCollector friendlyName="XPlat code coverage">
				<Configuration>
					<Format>lcov,cobertura</Format>
					<IncludeTestAssembly>false</IncludeTestAssembly>
					<CodeCoverage>
						<ModulePaths>
							<Exclude>
								<ModulePath>.*FluentAssertions.dll</ModulePath>
								<ModulePath>.*Moq.dll</ModulePath>
								<ModulePath>.*System.reactive.dll</ModulePath>
							</Exclude>
						</ModulePaths>
						<Sources>
							<Exclude>
								<Source>.*\\Reference.cs</Source>
							</Exclude>
						</Sources>
						<!-- Match attributes on any code element: -->
						<Attributes>
							<Exclude>
								<!-- Don't forget "Attribute" at the end of the name -->
								<Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
								<Attribute>^System\.Diagnostics\.DebuggerNonUserCodeAttribute$</Attribute>
								<Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
								<Attribute>^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$</Attribute>
							</Exclude>
						</Attributes>
					</CodeCoverage>
				</Configuration>
			</DataCollector>
		</DataCollectors>
	</DataCollectionRunSettings>
</RunSettings>

(note that this is the current state, I’ve been messing around with options to no avail)

Now when I run dotnet vstest supplying the full list of DLLs we want to check, I use the following command:

dotnet vstest /Diag:"c:\temp\diag.txt" /Settings:"C:\git\<solutiondir>\.azure\code-coverage.runsettings" /testadapterpath:"C:\Users\<user>\.nuget\packages\coverlet.collector\3.1.0\build\netstandard1.0" <listofdlls>

This runs and produces a coverage XML file, but for certain modules the coverage results are zero. Note: for certain modules, not for all! Overall coverage is reported to be 15% whereas Visual Studio when asked says 80%. So why is it missing things?

I’ve inspected the diagnostics output and found a number of Hits file not found lines which seem to correspond with the modules for which the output is not being shown. That makes sense, but why is it not finding these files? For other modules it shows Hit file deleted, which is probably what you’d want to see? I searched the other diag files for hints which might explain why there are no hit files, but couldn’t find any more leads. So I’m stuck now trying to find the root cause of this, any pointers where to look would be highly appreciated.

[whodurun4]

I’m experiencing similar behavior where I receive appropriate coverage reports on my development machine, but running in CI with an alpine-based image for a .net6.0 and coverlet collector.

Microsoft.NET.Test.Sdk 17.2.0 coverlet.collector 3.1.2

The cobertura file is generated, but only with empty sources and packages nodes.

[jsiegmund]

Sorry, here’s a ping to let you know I did not forget but also didn’t find the time to look into it. I’ll report back asap.