Create a .NET Standard version of the Engine

@rprouse commented on Mon Dec 14 2015

This needs design, but could be platform specific agents. If so, it will likely be blocked by #362.

---

@rprouse commented on Sat Jan 02 2016

In #1168, @CharliePoole said,

I’m not sure there is a problem here since this refers (in my mind anyway) to an engine running on the desktop. What we need to find out is whether an engine running under .NET 2.0/3.5 is able to analyze assemblies built for other targets. I don’t see why not, since they are not actually loaded. Consider that we are already analyzing .NET 4.5 assemblies. Of course, this needs to be tested on a VM that has only .NET 2.0 or 3.5 installed.

Based on my spikes into this so far, I think this is the way we need to go. The engine can inspect the test assembly and determine it’s target platform. We just can’t run it directly.

This is also mixed up with #677 and #362. My idea so far is as follows,

• The current NUnit3 engine inspects the test assembly in addition to the framework. It looks for the TargetFrameworkAttribute which will tell you for example if it is targeting .NETCore,Version=v5.0.
• If the target framework is a non-desktop framework, disallow --inprocess since we can’t run non-desktop targets. We might also want to disallow --process=Single or we make sure all test assemblies target the same platform.
• Modify TestAgency to allow it to launch platform specific agents and communicate with them in a manner other than .NET remoting. This is #362. If we are running those agents on the desktop machine, we could even capture Console.In/Out for communications.

The platform specific agents are where decisions still need to be made. It would be nice to have a portable mini-engine (#677) that is capable of loading and running the portable framework. Like the full framework, this would allow us to run test no matter which version of the NUnit 3 framework is used. It would likely be a stripped down version of the engine that can only load from the agent directory and is only intended to be used by agents. This might even boil down to a portable IFrameworkDriver and some supporting classes?

A simplified and not mutually exclusive approach might be that the agents are actually the nunitlite version of the tests. We could modify nunitlite to run directly, or be able to communicate with the engine.

---

@CharliePoole commented on Sat Jan 02 2016

I think we’re pretty much on the same page at a high level. Some details…

• Can we count on TargetFrameworkAttribute being reliably set?
• We can’t precisely disallow all those options, although we might be able to in some cases. Basically, we can only look at explicit options, but if the defaults are used, we have to wait until we analyze the assembly and then reflect an error back to the runner.
• I think we still have two very distinct options for running portable and device-based tests: a scaled down engine versus a scaled up driver. Neil has been pushing us strongly toward an engine for CF but I’m not sure that’s the easiest approach. At some point, we have to spike both approaches. I think the easiest way to do that initially is right on the desktop itself.

Charlie

---

@rprouse commented on Mon Jan 04 2016

Can we count on TargetFrameworkAttribute being reliably set?

I have tested quite a few different assemblies and have not found it unset except for .NET 2.0 assemblies. I think that we can safely assume that if it is not set, it targets a full .NET framework. I assume that these days with so many potential targets, it needs to be set.

I think we still have two very distinct options for running portable and device-based tests: a scaled down engine versus a scaled up driver. Neil has been pushing us strongly toward an engine for CF but I’m not sure that’s the easiest approach. At some point, we have to spike both approaches. I think the easiest way to do that initially is right on the desktop itself.

I installed the .NET Portability Analyzer extension for Visual Studio and analyzed the engine against the platforms that we want to support. Based on what is not available, I think approaching it from the driver end is probably the best option. At a minimum, we need to be able to load the framework in a version independent way, explore or run the tests and communicate the results. The majority of the functionality of the engine is used in the startup process, not in the agents themselves.

---

@xied75 commented on Wed Feb 17 2016

👍 to track.

---

@rprouse commented on Wed Feb 17 2016

@xied75, for future reference, you can just click the subscribe button at the bottom of the toolbar on the left. That way, all other subscribers don’t get an email 😉

---

@xied75 commented on Wed Feb 17 2016

Well this is in fact a new trend on GitHub, since GitHub refuses to add the feature to allow one see what he/she subscribed, so now people do +1 in order to find what they want track.

---

@rprouse commented on Fri Jun 10 2016

I am going to move this out of 3.4. The new dotnet-test-nunit will run .NET Core and UWP apps and the promise of netstandard may change the way that we approach this. I want to see how things go after the release of .NET Core before committing to major changes in the engine.

---

@CharliePoole commented on Fri Jun 10 2016

That makes perfect sense.

[rprouse]

@CharliePoole most people who develop .NET Core on other platforms these days don’t have Mono installed. .NET Core includes command line tools to create new projects, build, debug and run .NET Core and .NET Standard projects. Mono is not used and not needed.

To edit the files, most people use Visual Studio Code, not MonoDevelop. VS Code is a native application that does not require .NET to run. VS Code uses OmniSharp to provide intellisense which also works with other code editors.

I expect that on Linux and Mac, Mono will only be used for desktop apps and I am not seeing much going on there. Even Miguel recommends against using Mono for server side code which is what most people are using .NET Core for (ASP.NET MVC websites and REST APIs). Because of that, most Linux/Mac .NET developers these days don’t need or want Mono installed. The only reason I have it installed on any machines is to test NUnit 😄

Even on Windows, .NET Core has nothing to do with the Full .NET Framework. It does not compile, debug or run using the framework and cannot run within the full CLR. It runs in the CoreCLR.

I should also mention that testing is built into the Core as a first class citizen now. It is a command just like build, debug or run and is the standard way of running unit tests in .NET Core. That is why we should support that scenario.

I will see if I can come up with some diagrams to explain how this version of the engine would be used.

[rprouse]

I have been working on this and wanted to update people on my plans and progress so that they can provide feedback.

The VS Adapter, Xamarin runner and dotnet test all need a version of the engine that can load and run any .NET Standard based tests. For example, see nunit/nunit3-vs-adapter#297.

All of these runners run in-process, they do not launch an agent. Visual Studio inspects the assembly and launches a platform specific test host (desktop, UWP, .NET Core, etc) which then attempts to load our platform specific version of the VS adapter. (See https://github.com/Microsoft/vstest-docs/tree/master/RFCs).

Because of this, I am concentrating on the parts of the API/Engine that run in-process for the first pass. This will allow us to move forward with the Visual Studio Adapter and drop much of the duplicate code in the Xamarin runner. Future work could allow the desktop version of the engine to launch platform specific agents (.NET Core, Xamarin Android, etc), but I think we should hold off on that until we know more about how the new dotnet CLI plays out. AFAIK, Microsoft’s plan is to have the dotnet test command be the standard way to run tests from the command line. If that takes off, then we may not need support in our console runner.

The Engine API converted easily. The only change to the API was dropping AppDomain from one of the method signatures. AppDomain is not supported in .NET Standard.

Conversion of the Engine is nearly done and going well. The only thing I am having trouble with is Assembly resolution in ProvidedPathsAssemblyResolver. Because there are no AppDomains, we will also run into problems with projects that have tests using different versions of NUnit.

We will need to decide if we want to have .NET Standard versions of any of our extensions. Potential candidates would likely be,

• NUnit V2 result writer
• NUnit project loader
• TeamCity event listener

If we do convert them, the NuGet packages will need to be modified to include the various targets and our extension search will need to be updated to find the correct platform versions.

If anyone wants to see the work in progress, I can create a pull request for review.