Stryker Randomly Reports NoCoverage for all mutants

Describe the bug Stryker randomly reports NoCoverage for all non-excluded mutants.

When running against multiple projects (individually) I can almost always reproduce the problem in at least one of them by specifying max-concurrent-test-runners with a value other than the default. I was originally going to say that was necessary to reproduce the issue, but I am still getting random (though less frequent) runs reporting NoCoverage without it so concurrency may be a red herring.

These runs then break my automated builds–I have a script that collects the output from each project and combines them into a single report which fails due to missing result files.

Logs I have logs but I’d prefer not attaching them publicly as they look to contain our source code. Can I provide these directly to someone? If not, I’ll see what I can do about redacting them.

Expected behavior Stryker should detect the same coverage/no-coverage status every run.

Desktop (please complete the following information):

• OS: Windows 10 and Ubuntu 18.04
• Type of project: Core
• Framework Version: 3.1/5.0 (see below)
• Stryker Version: 0.22.9

Additional context I have a solution with five projects in it. The projects-under-test are multi-targeted .NET5.0 and .NETStandard2.1 (and in one instance, .NETCOREAPP3.1 in lieu of .NETStandard). The Test projects themselves are multi-targeted 3.1/5.0. Each project has a configuration file that looks like this:

{
	"stryker-config": {
		"reporters": [ "progress", "html", "json" ],
		"max-concurrent-test-runners": 4,
		"coverage-analysis": "perTest",
		"project-file": "The.Project.Name.csproj",
		"language-version": "CSharp9",
		"mutation-level": "Advanced",
		"ignore-methods": [ "ToString", "ConfigureAwait", "*Exception.ctor", "Log*" ],
		"excluded-mutations": [	"string" ],
		"mutate": [ "!**/Some.Exclusions.cs" ]
	}
}

For two of the test projects, specifying project-file is necessary as there are a handful of tests that validate some interactions between two of the libraries (unavoidable). For consistency I specify the project-file in all configuration files.

I run Stryker for all of the test projects, individually of course. With the above configuration I am pretty much guaranteed that at least one of the test projects will report that all tests are NoCoverage. It’s not always the same project(s). When running in my build pipeline it might be the first, it might be the third. Because the pipeline fails at the first missing result file it’s hard to say what the remaining projects would do.

If I remove max-concurrent-test-runners and rely on the default, I get a smooth run more often then not–but not always. Re-running the “failed” executions with the very same configuration will usually produce a completed run.

This is all happening in my build pipelines. I sometimes experience the same behavior locally but only when overriding the default concurrency (I have not yet experienced it locally using the defaults).

Some other notes:

• My PC is Windows 10 with .NET Core 3.1, .NET 5.0 and 6.0 Preview installed.
• My Build agents are Ubuntu 18.04 and have 3.1/5.0
  • For some reason Stryker always goes single-threaded on the Linux VMs, despite having 8 cores, which is why I was specifying the concurrency value
• Every run of Stryker (locally and on VMs) also produces Warnings/FileNotFoundExceptions for
  • Microsoft.CodeAnalysis.NetAnalyzers, Version=5.0.4.22903
  • Microsoft.CodeAnalysis.Workspaces, Version=3.3.0.0

My solution structure looks like the following:

\Solution.sln
\src
    \ProjectA
    \ProjectB
\test
    \ProjectA.Tests
    \ProejctB.Tests

[pinkfloydx33]

Haven’t seen it happen; I’m on Stryker 1.x now and .NET 6 on that specific project which I’m sure makes some sort of difference

[dupdob]

@pinkfloydx33 : if you still have this problem, can you please try again with the latest release? Several bug fixes have been implemented and the situation should have improved