Restore dotnet test functionality by separating it from IDE concerns

This is in response to the breakage described by the xUnit team here: https://github.com/dotnet/cli/issues/3114#issuecomment-264662326

Summary

By separating IDE integration from dotnet test, test frameworks’ own features can be made available again to end users, and test frameworks can continue to be as lean as they always have been.

Background

Before preview 3, dotnet test allowed for both CLI mode and IDE mode test running. Test frameworks would provide an executable with a discoverable name, “dotnet-test-framework.exe”.

When invoked from the command line, dotnet test would find and run the framework’s exe once for each target framework, and would use the overall results of each run to decide whether the overall run succeeded or failed. dotnet test would use the special positional argument “–” to avoid any ambiguity of command line arguments, passing them along to the test framework so that the test framework’s own features were all exposed to the end user.

When invoked by the IDE for Test Explorer, the same framework-provided exe would be located and used, this time in a “design time” mode. Test Explorer naturally needs a lot more information from the test framework in order to do its job. There’s a discovery phase that is irrelevant in CLI mode, and a data model for the discovery phase and for the execution phase that gives the IDE enough information to show everything that happened.

Before preview 3, this all worked using a series of JSON messages passed back and forth between dotnet test and the framework-provided exe.

There were a few downsides to this approach. Test framework authors essentially had to copy and paste a lot of each other’s code to do the JSON message passing correctly, and they started taking on JSON.NET dependencies that might trample the JSON.NET dependency of the system under test.

With the preview 3 tooling, that whole approach is now gone. Instead, both CLI mode and IDE mode test running happens through the tried-and-true IDE integration APIs that existed before .NET Core and the dotnet tooling. While that’s just fine for Test Explorer, it means that all CLI test runs happen through all of that infrastructure. The most serious issue is that command line arguments no longer pass through, meaning that end users can’t use dotnet test for any framework-specific features, even fundamental ones like the XML reporting understood by many CI systems.

Recommendation

It might be tempting to work around the command line argument pass-thru issue by extending the Test Explorer interfaces to include a string[ ] argument, but I think that just takes an earlier misstep and cements it.

Instead, let’s consider that CLI test running and IDE test running are simply different things, and always have been in all of the .NET test frameworks:

1. Continue with the preview 3 tooling approach for IDE integration. The existing API for integrating with Test Explorer works, and is a natural fit for projects like NUnit and xUnit who are trying to take their existing code and adapt it to the .NET Core world. It’s familiar, it works, and porting to .NET Core isn’t all that difficult for this component.
2. Switch CLI mode test runs to have nothing to do with IDE integration.

Point 2 is worth clarifying. Picture the previous tooling that used JSON message passing between processes. All of that message passing was really a result of the IDE mode concerns, in which a lot of data does need to be communicated back for display in the UI. Without IDE concerns, the discovery phase and all its message passing goes away, as does all the rest of the message passing.

What would be left? What would the dotnet test CLI tool look like? It would become pretty trivial:

1. Ensure the target project(s) are built, like it already did.
2. Loop through the test project’s target frameworks (net45, netstandard1.3…), like it already did.
3. For each target framework, find the dotnet-test-framework assembly, like it already did.
4. Start up a Process (passing through command line args) for that assembly and WaitForExit. No need for the previous socket and threading concerns. New Process, WaitForExit.
5. Consider the process’s return code. If it’s 0, success. Otherwise, it failed.

That’s it. No more information needs to pass between the two processes in CLI test runs. Let the test frameworks own all other output in CLI mode, as they always have. dotnet test would save us from having to do the “target framework loop” ourselves, but wouldn’t bring along any IDE baggage like detailed message passing, *trx file generation/processing, etc. All the current work to make IDE integration work with the tried and true Visual Studio interfaces could continue unharmed.

Each test framework could remain lean and featureful without having to “pay the Test Explorer tax”.

[citizenmatt]

I just want to add that there are more IDE runners than just Visual Studio - we have both ReSharper and Rider to think about. And we want to continue offering existing functionality - running multiple or individual tests, debugging, profiling, and code coverage.

(And I have to again voice some frustration here. While we are in conversation with the responsible team, we’re still waiting for details - high level overview, API specs, roadmap, capabilities, etc. It’s not a good sign that an open source project can ship such a breaking change without releasing the source, documenting anything or even discussing it in the open before hand.)

[derwasp]

@RehanSaeed try this: dotnet test myproj.csproj --logger "trx;LogFileName=c:\logfile.trx"