Integrating with the test runner

By far the next biggest performance improvement is integrating with the test runner. Right now, we support the dotnet test command. This spins up a new dotnet process. There is a lot of overhead here before dotnet even starts executing the tests:

• OS Allocation of process resources
• dotnet allocation of heap + other overhead
• Loading of all assemblies from disk
• Exiting of the process.

It would be great if we can host the child process ourselves. In this process we can load everything once and reuse that process to run tests (each time with a different ACTIVE_MUTATION environment variable). Having the overhead just once will dramatically improve performance.

In Stryker (for JS) this is done with a test runner plugin system. The command test runner is just one of the available implementations. Other examples are: MochaTestRunner (stryker-mocha-runner) and the JestTestRunner (stryker-jest-runner). Something similar here would be great to have. Not sure if it needs to be it’s own separate dependency.

If you open a solution in Visual Studio, you can see the tests on the left. It doesn’t matter if they are implemented in MSTest, NUnit or XUnit. This involves some kind of TestAdapter interface. Maybe we can leverage this api as well, so we only need to implement it once, instead of for each test runner.

[rouke-broersma]

PR to add vstest.console.dll support to vstest client library has been submitted: https://github.com/Microsoft/vstest/pull/1893

The change worked on my system to start vstest using dotnet from stryker and tests were discovered and run by stryker through vstest.

[rouke-broersma]

Yes, I am working on this. See https://github.com/stryker-mutator/stryker-net/tree/183-vstest-integration for working POC (windows only) of vstest integration. I am currently working on improving the vstest client library (provided by microsoft) as it does not support vstest.console.dll (xplat) at the moment. We cannot integrate vstest until that is fixed. See bug with vstest team: https://github.com/Microsoft/vstest/issues/1887 I have started the work here: https://github.com/Mobrockers/vstest/tree/1187_add_support_for_xpat_vstest_console_in_translationlayer But this still needs work, vstest.console.dll is not yet successfully started.