Solution with netcoreapp3 and net472 projects

Description

My projects have both kind of projects: net472 and netcore3. Very simple sample is here: vstestissue.zip

Steps to reproduce

In folder with vstestissue.sln run:

msbuild /t:restore;build
vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll

Expected behavior

  √ TestMethodnetcore [2ms]
  √ TestMethodnetfw [3ms]

Test Run Successful.
Total tests: 2
     Passed: 2
 Total time: 0,5982 Seconds

All tests runes successful. vstest group all binaries by framework and run group in suitable runtime.

Actual behavior

vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.0 and platform X86. Following DLL(s) do not match framework/platform settings.
netfw.dll is built for Framework .NETFramework,Version=v4.7.2 and Platform AnyCPU.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][.....\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '.....\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

  √ TestMethodnetfw [3ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 0,3968 Seconds

If I change order:

vstest.console.exe  netcore\bin\Debug\netcoreapp3.1\netcore.dll netfw\bin\Debug\net472\netfw.dll
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.0 and platform X86. Following DLL(s) do not match framework/platform settings.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
netfw.dll is built for Framework .NETFramework,Version=v4.7.2 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][...\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '...\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

[MSTest][Discovery][...\netfw\bin\Debug\net472\netfw.dll] Failed to discover tests from assembly ...\netfw\bin\Debug\net472\netfw.dll. Reason:Could not load file or assembly 'netfw' or one of its dependencies. The system cannot find the file specified.
No test is available in ...\netcore\bin\Debug\netcoreapp3.1\netcore.dll ...\netfw\bin\Debug\net472\netfw.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

echo %errorlevel% 0

If I add framework netcore3:

vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll "/framework:.NETCoreApp,Version=v3.1"

Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETCoreApp,Version=v3.1 and platform X64. Following DLL(s) do not match framework/platform settings.
netfw.dll is built for Framework .NETFramework,Version=v4.7.2 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
Unable to find ...\netfw\bin\Debug\net472\netfw.deps.json. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk".
  √ TestMethodnetcore [2ms]

Test Run Aborted.
Total tests: Unknown
     Passed: 1
 Total time: 0,7384 Seconds

echo %errorlevel%
1

If I change framework to 4.7.2 I get another behavior:

vstest.console.exe  netcore\bin\Debug\netcoreapp3.1\netcore.dll netfw\bin\Debug\net472\netfw.dll  "/framework:.NETFramework,Version=v4.7.2"
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.7.2 and platform X86. Following DLL(s) do not match framework/platform settings.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][...\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '...\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

[MSTest][Discovery][...\netfw\bin\Debug\net472\netfw.dll] Failed to discover tests from assembly ...\netfw\bin\Debug\net472\netfw.dll. Reason:Could not load file or assembly 'netfw' or one of its dependencies. The system cannot find the file specified.
No test is available in ...\netcore\bin\Debug\netcoreapp3.1\netcore.dll ...\netfw\bin\Debug\net472\netfw.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

echo %errorlevel% 0

if I change order of files and leave .net742:

vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll   "/framework:.NETFramework,Version=v4.7.2"
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.7.2 and platform X86. Following DLL(s) do not match framework/platform settings.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][...\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '...\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

  √ TestMethodnetfw [3ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 0,4156 Seconds

echo %errorlevel% 0

Now I have to somehow separate the binaries myself and take two steps during CI. I don’t want add knowledge about type of projects (and names) to the CI, so I separate by analyzing the targetframeworkAttribute of the binary files and run each of group files in vstest.console.exe with suitable framework.

Environment

OS: win10: 1803…2004, winserver 2019 vstest.console: 16.4, 16.5.0-preview-20200110-02

AB#1494522

[gormac]

Could you share how you work around the problem now? For me and others to see 🙂

The workaround is as simple as running vstest twice, for each platform with filters applied. The (file) filter for the .NET Core tests currently is ‘**/netcoreapp3.1/*Tests.dll’. This prevents vstest from breaking on libraries that target .NET Framework. It doesn’t feel like a very future proof workaround.

[nohwnd]

The ability to run multiple TFMs and multiple architectures at the same time will be shipped in 17.3 (there are already previews available). You should avoid using the platform and framework options, because they force a single unified platform/framework on all the dlls, instead of letting autodetection figure out how is each dll built.