FTL: The name 'AppDomain' does not exist in the current context (Source code: AppDomain)

Describe the bug A clear and concise description of what the bug is.

Logs For us to best assist you in diagnosing the problem we would ask you to run stryker with the option --log-file and to attach the resulting logs to the issue.

This is broken?

~ dotnet stryker -f stryker.log
Specify --help for a list of available options and commands.

Version: 0.13.0 (beta)

[14:34:44 INF] The project /Users/manuelriezebosch/git/Unmockable/Unmockable.Intercept/Unmockable.Intercept.csproj will be mutated
[14:34:46 INF] Started initial build using dotnet build
[14:34:48 INF] Initial build successful
[14:34:54 INF] Using testrunner VsTest
[14:34:54 INF] Total number of tests found: 48
[14:34:54 INF] Initial testrun started
[14:34:55 INF] Using 7536 ms as testrun timeout
[14:34:58 WRN] Stryker.NET encountered an compile error in Stryker.Core.InjectedHelpers.MutantControl.cs with message: The name 'AppDomain' does not exist in the current context (Source code: AppDomain)
[14:34:58 WRN] Safe Mode! Stryker will try to continue by rolling back all mutations in method. This should not happen, please report this as an issue on github with the previous error message.
[14:34:58 FTL] Stryker.NET could not compile the project after mutation. This is probably an error for Stryker.NET and not your project. Please report this issue on github with the previous error message.
[14:34:58 ERR] An error occurred during the mutation test run 
System.ApplicationException: Internal error due to compile error.
   at Stryker.Core.Compiling.RollbackProcess.RemoveMutantIfStatements(SyntaxTree originalTree, ICollection`1 diagnosticInfo, Boolean devMode)
   at Stryker.Core.Compiling.RollbackProcess.Start(CSharpCompilation compiler, ImmutableArray`1 diagnostics, Boolean devMode)
   at Stryker.Core.Compiling.CompilingProcess.TryCompilation(MemoryStream ms, CSharpCompilation compilation, EmitResult previousEmitResult, Boolean devMode, Int32 retryCount)
   at Stryker.Core.Compiling.CompilingProcess.Compile(IEnumerable`1 syntaxTrees, MemoryStream ms, Boolean devMode)
   at Stryker.Core.MutationTest.MutationTestProcess.Mutate()
   at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options, IEnumerable`1 initialLogMessages)
[14:34:58 INF] Time Elapsed 00:00:15.8082994

Expected behavior A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

• OS: Windows & MacOS
• Type of project: core
• Framework Version 3.0.100 (SDK)
• Stryker Version: 0.13

Additional context https://github.com/riezebosch/Unmockable

[dupdob]

Quick update: I reproduce the coverage issue and I have started analysing it. The problem appears to happen during the coverage analysis phase, where some VsTest errors happen, invalidating data capture for the affected tests. Next step is to understand what triggers this errors and fix it.

[dupdob]

I have completed the analysis regarding the coverage situation.

Issue: Significant discrepancy in mutation testing result when disabling coverage analysis on Unmockable

All mutants (37 at time of testing) are killed when turning coverage analysis off whereas 6 are uncovered and 3 survived with coverage analysis on.

Cause: Unexpected impact of mutant control logic

Unmockable extensively uses the System.Linq.Expressions.Expression class in method signature. StrykerMutator.Net relies on an in-place mutation for expressions, using ternary operators to enable them when needed. Therefore some mutated expressions are not evaluated at runtime, but simply passed as an expression tree to Unmockable code. As a result, mutant in non evaluated expressions:

• are invisible during coverage analysis ⇒ flagged as not covered
• can’t be selectively activated ⇒ can’t be tested
• result mutated code that is active during every test sessions ⇒ Stryker.Mutator results are wrong with or without coverage analysis.

To confirm that last statement, one simply has to manually introduce a mutant and run the code. For example, this mutant breaks no test and should be marked as survived

namespace Unmockable.Matchers
{
  internal static class ArgMatcherFactory
  {
    public static IArgumentMatcher? Create(Expression arg) => arg switch
      {
          MethodCallExpression call when call.Method.DeclaringType == typeof(Arg) => call.Method.Name switch
          {
              nameof(Arg.Ignore) => (IArgumentMatcher)new IgnoreArgument(),
              // here we have replaced Arguments.Single() with Arguments.SingleOrDefault()
              nameof(Arg.Where) => new LambdaArgument(call.Arguments.SingleOrDefault()),
              _ => throw new InvalidOperationException()
          },
          _ => null
      };
   }
}

But it is marked as ‘not covered’ with coverage analysis and as ‘killed’ when coverage analysis is off. Please note that having it marked as ‘killed’ means it is very likely that other mutants are flagged as ‘killed’ while they are actual survivors, invalidating the test session’s results.

Workaround

The only workaround is to use the ignore methods option (see #646) to disable mutation for each call to method expecting System.Linq.Expressions.Expression as parameter(s).

Fix

At this time, there is no clear and fast approach to fix this issue. Leads are:

1. Identify offending method(s) and automatically disable related mutation. Can’t be done as of now, as Stryker.Net does not implement the required call resolution logic to reliably identify those calls
2. Offer an option to only use if statements to control mutation logic. Difficult to do as it requires reliable identification of enclosing statements, it would result in bloated mutated assemblies due to the extra duplication and it is not universal (usage of expression body instead of statement)
3. A mixing 1 and 2 would limit the bloating (only use ifs when necessary) and handle the expression body problem by disabling mutation for those
4. Adjust mutant control logic to make it transparent for System.Linq.Expressions.Expression. I have no idea how to achieve that, if it is possible.

Other findings

This analysis clearly revealed that several logging messages where improperly formatted and/or where not providing the desired informations. I will open a PR that will fix all identified issues.

Original issue

Now that I can reproduce the issue, I will have a look at the problem with ‘AppDomain’