dotnet test running assemblies in parallel (when mixing core and framework test assemblies)

I have an issue where “dotnet test” is running assemblies in parallel. I have tried adding [NonParallelizable] and [assembly: LevelOfParallelism(1)] and also running with the dotnet test CLI option -p:ParallelizeTestCollections=false none of which has helped.

I have:

Solution \ MyLibrary.csproj (netstandard2.0) \ Tests461 (net461) \ TestsCore2 (netcoreapp2.2)

The nuget dependencies of both test csproj files are:

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.2.0" />
    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.15.1" />
    <PackageReference Include="NunitXml.TestLogger" Version="2.1.41" />
  </ItemGroup>

When I run the tests using

dotnet test

I get the following logged:

Core SetUp 688503812
Core Enter Test 688503812
461 SetUp 688506609
461 Enter Test 688506609
Core Exit Test 688506812
461 Exit Test 688509625

It even looks like dotnet test is firing up 2 instances of the tester process (which don’t respect each other’s parallelization):

PS E:\MyLibrary> dotnet test
Test run for E:\MyLibrary\TestsCore2\bin\Debug\netcoreapp2.2\TestsCore2.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0-preview-20190715-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run for E:\MyLibrary\Tests461\bin\Debug\net461\Tests461.dll(.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0-preview-20190715-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.0100 Seconds

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.0288 Seconds
PS E:\MyLibrary>

I have built a simple 3 project solution that reproduces the problem if it would help I can provide.

Other things I have tried:

dotnet test -- NUnit.NumberOfTestWorkers=1

[mikeharder]

@tznind: See a workaround I posted here: https://github.com/dotnet/cli/issues/11017#issuecomment-564311592. I verified this works in your minimum repro:

$ dotnet test dirs.proj -p:BuildInParallel=false
Test run for D:\Git\DotnetTestParallelExample\Tests461\bin\Debug\net461\Tests461.dll(.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.3995 Seconds
Test run for D:\Git\DotnetTestParallelExample\TestsCore2\bin\Debug\netcoreapp2.2\TestsCore2.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.7372 Seconds
 
$ type C:\temp\output.txt
461 SetUp -1944219765
461 Enter Test -1944219750
461 Exit Test -1944216750
Core SetUp -1944214390
Core Enter Test -1944214375
Core Exit Test -1944211375

[smchan514]

For anyone still looking for a solution, there is the command line option -m:1 which specifies the maximum number of concurrent processes to use when building.

dotnet test -m:1 MyUnitTests.sln

Ref:

1. https://github.com/dotnet/sdk/issues/10178
2. MSBuild command-line reference – Switches