Loading a multi-targeting csproj using nunit3-console gives NRE exception

I have the following project file (one of 9 in this .sln, they all are multitargeting)

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp2.0;net462</TargetFrameworks>
    <AssemblyName>AsyncLinqTests</AssemblyName>
    <RootNamespace>LinqTestsAsync</RootNamespace>
    <DebugType Condition=" '$(TargetFramework)' == 'net462'">Full</DebugType>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
    <PackageReference Include="NUnit" Version="3.11.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.11.0" />
    <PackageReference Include="System.Data.SqlClient" Version="4.5.1" />
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' == 'net462'">
    <Reference Include="$(SolutionDir)RuntimeLibraries\SD.LLBLGen.Pro.DQE.SqlServer.dll" />
    <Reference Include="$(SolutionDir)RuntimeLibraries\SD.LLBLGen.Pro.ORMSupportClasses.dll" />
    <PackageReference Include="SD.Tools.OrmProfiler.Interceptor.NET45" Version="1.5.20150715" />
    <Reference Include="System.Transactions" />
    <Reference Include="System.Windows.Forms" />
    <Reference Include="$(SolutionDir)ExpressionTreeVisualizer\ExpressionTreeVisualizer\bin\Debug\ExpressionTreeVisualizer.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\InheritanceTwo.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\InheritanceTwo.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\InheritanceTwo.SelfServicing.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\NW26Async.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\NW26Async.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\NW26Async.SelfServicing.dll" />
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp2.0'">
    <Reference Include="$(SolutionDir)RuntimeLibraries\NetStandard2.0\SD.LLBLGen.Pro.DQE.SqlServer.dll" />
    <Reference Include="$(SolutionDir)RuntimeLibraries\NetStandard2.0\SD.LLBLGen.Pro.ORMSupportClasses.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\InheritanceTwo.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\InheritanceTwo.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\InheritanceTwo.SelfServicing.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\NW26Async.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\NW26Async.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\NW26Async.SelfServicing.dll" />
  </ItemGroup>
</Project>

Using Rider (latest) or VS 2017 and this .sln runs the tests fine (with the nunit adapter).

However using: nunit3-console AllUnitTests.sln

gives:

NUnit Console Runner 3.9.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    AllUnitTests.sln

System.ArgumentException : Invalid project file format: AsyncLinqTests.csproj
  ----> System.NullReferenceException : Object reference not set to an instance of an object.

--ArgumentException
Invalid project file format: AsyncLinqTests.csproj
   at NUnit.Engine.Services.ProjectLoaders.VSProject.ThrowInvalidFormat(String projectPath, Exception e)
   at NUnit.Engine.Services.ProjectLoaders.VSProject.Load()
   at NUnit.Engine.Services.ProjectLoaders.VSSolution.Load()
   at NUnit.Engine.Services.ProjectLoaders.VisualStudioProjectLoader.LoadFrom(String path)
   at NUnit.Engine.Services.ProjectService.ExpandProjectPackage(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.ExpandProjects(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.EnsurePackagesAreExpanded(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.EnsurePackagesAreExpanded(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.InitializePackage()
   at NUnit.Engine.TestEngine.GetRunner(TestPackage package)
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
   at NUnit.ConsoleRunner.Program.Main(String[] args)
--
NullReferenceException
Object reference not set to an instance of an object.
   at NUnit.Engine.Services.ProjectLoaders.VSProject.TryLoadDotNetCoreProject()
   at NUnit.Engine.Services.ProjectLoaders.VSProject.Load()

Nunit3-console is the latest (downloaded it 2 hours ago), used the .msi installer.

Loading the csproj instead of the sln gives the same error. Running nunit3-console with the assembly works, but is less ideal as there are multiple targets and that would be more work to set up (I’m looking at running the tests on the command line as running the tests inside an IDE is slow: 3800 tests over 2 platforms take over 8 minutes combined)

[FransBouma]

This turned out to be super simple 😃 dotnet test is actually using msbuild to run the tests. So it’s configurable using msbuild parameters:

dotnet test AllUnitTests.sln -v q -m:1

limits it to 1 testrunner at a time, using the -m:n flag.

Hope this helps other people with the same problem 😃 An additional bonus here is that it runs the tests on all target platforms in the csprojs so I have 2 platforms per csproj and it runs all tests, using nunit on all targeted platforms, on the command line. Only downside is the configuration of e.g. the output. The nunit console runner has more options to configure things.

[CharliePoole]

You might consider creating an NUnit project that tracks the test assemblies in your solution. A few hints:

1. A config in an NUnit project is an arbitrary string that calls out a specific set of test assemblies. The obvious way to use it is to define configs that match the solution, but a more creative approach would be possible in your case.

2. Relative assembly paths in the project file are based on the location of the .nunit file itself. Especially important to know if you are creating the XML by hand.

3. NUnit will attempt do whatever you tell it to, whether it makes sense or not, so don’t try to load incompatibly targeted assemblies into the same AppDomain or Process.

4. You are still subject to the limitations of the console runner in terms of which targets are supported.