[MSTest][Discovery] Unable to load types from the test source ... Could not load file or assembly

• Windows 10 Enterprise x64
• Visual Studio Enterprise 2022 (64 bit) Version 17.5.4
• DemoProject.dll: Project under test targeting .Net standard 2.0
• DemoTestBase.dll: Test helper project targeting .Net standard 2.0
• DemoUnitTestProject.dll: Unit test project targeting .Net 6 and .Net framework 4.7.2. Having 2 unit test classes: UnitTest1.cs, UnitTest2.cs, with UnitTest1 inhering from DemoTestBase, but not UnitTest2.
• These 3 projects are in the same solution.
• The unit test project is referencing the other 2 projects using project reference.
• Using a .runsettings file to resolve assembly directory path.

Assembly output structure

C:\MyDev\Sources\bin\DemoProject.dll
C:\MyDev\Sources\bin\DemoTestBase.dll
C:\MyDev\Sources\unittest\DemoUnitTestProject.dll (this is .Net framework 4.7.2)
C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll (this is .Net 6)

When I do a Clean and Rebuild on the unit test project, I see below error message in the Tests pane (with logging level = Trace).

[Warning] [MSTest][Discovery][C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll] Unable to load types from the test source 'C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'DemoTestBase, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

This is the complete trace

[28/04/2023 11:56:07.910 AM]  Interrupt: Enqueueing WaitForBuildOperation
[28/04/2023 11:56:07.910 AM]  Enqueue operation 'WaitForBuildOperation', hashcode:42412516 
[28/04/2023 11:56:07.910 AM]  Operation left in the the queue: 1
[28/04/2023 11:56:07.910 AM]  	'WaitForBuildOperation', hashcode:42412516
[28/04/2023 11:56:07.910 AM]  

[28/04/2023 11:56:07.910 AM]  Operation Dequeue : 'WaitForBuildOperation'
[28/04/2023 11:56:09.763 AM]  test container discoverer executor://projectoutputcontainerdiscoverer/v1, discovered 2 containers
[28/04/2023 11:56:09.763 AM]  Containers from 'Microsoft.VisualStudio.TestWindow.Client.TestContainer.ProjectOutputContainerDiscoverer' :
[28/04/2023 11:56:09.763 AM]  	C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll
[28/04/2023 11:56:09.763 AM]  	C:\MyDev\Sources\unittest\DemoUnitTestProject.dll
[28/04/2023 11:56:09.763 AM]  Enqueue operation 'DiscoverSelectedOperation', hashcode:495756 
[28/04/2023 11:56:09.763 AM]  Operation left in the the queue: 1
[28/04/2023 11:56:09.763 AM]  	'DiscoverSelectedOperation', hashcode:495756
[28/04/2023 11:56:09.763 AM]  

[28/04/2023 11:56:09.763 AM]  Operation Dequeue : 'DiscoverSelectedOperation'
[28/04/2023 11:56:09.764 AM]  Start updating 2 containers.
[28/04/2023 11:56:09.764 AM]  Updating container C:\MyDev\Sources\unittest\DemoUnitTestProject.dll.
[28/04/2023 11:56:09.777 AM]  Successfully refreshed container: C:\MyDev\Sources\unittest\DemoUnitTestProject.dll
[28/04/2023 11:56:09.777 AM]  Updating container C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll.
[28/04/2023 11:56:09.779 AM]  Successfully refreshed container: C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll
[28/04/2023 11:56:09.779 AM]  Completed updating containers.
[28/04/2023 11:56:09.809 AM]  ========== Starting test discovery ==========
[28/04/2023 11:56:09.810 AM]  File timestamp changed from 28/04/2023 11:13:31 AM to 28/04/2023 11:56:09 AM for C:\MyDev\Sources\unittest\DemoUnitTestProject.dll
[28/04/2023 11:56:09.810 AM]  File timestamp changed from 28/04/2023 11:13:31 AM to 28/04/2023 11:56:09 AM for C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll
[28/04/2023 11:56:09.811 AM]  Grouped C:\MyDev\Sources\unittest\DemoUnitTestProject.dll
C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll : (, False)
[28/04/2023 11:56:09.811 AM]  Test discovery settings for C:\MyDev\Sources\unittest\DemoUnitTestProject.dll,C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll:
 <RunSettings>
  <!-- MSTest adapter -->
  <MSTest>
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <CaptureTraceOutput>False</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>True</DeploymentEnabled>
    <AssemblyResolution>
      <Directory path="..\..\" includeSubDirectories="true" />
    </AssemblyResolution>
  </MSTest>
  <RunConfiguration>
    <ResultsDirectory>C:\MyDev\Sources\DemoProject\TestResults</ResultsDirectory>
    <SolutionDirectory>C:\MyDev\Sources\DemoProject\</SolutionDirectory>
    <MaxCpuCount>0</MaxCpuCount>
    <EnvironmentVariables>
      <VSTEST_BACKGROUND_DISCOVERY>1</VSTEST_BACKGROUND_DISCOVERY>
    </EnvironmentVariables>
    <CollectSourceInformation>False</CollectSourceInformation>
  </RunConfiguration>
</RunSettings>.
[28/04/2023 11:56:09.966 AM]  Logging TestHost Diagnostics in file: C:\Users\leef109860\AppData\Local\Temp\TestPlatformLogs\2023_04_28_11_07_32_26100\logs.host.23-04-28_11-56-09_81326_12.txt
[28/04/2023 11:56:10.040 AM]  Logging TestHost Diagnostics in file: C:\Users\leef109860\AppData\Local\Temp\TestPlatformLogs\2023_04_28_11_07_32_26100\logs.host.23-04-28_11-56-09_81425_16.txt
[28/04/2023 11:56:10.153 AM] [Warning] [MSTest][Discovery][C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll] Unable to load types from the test source 'C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'DemoTestBase, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

[28/04/2023 11:56:10.860 AM]  ========== Test discovery finished: 3 Tests found in 1.1 sec ==========
[28/04/2023 11:56:12.293 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\obj\Debug\DemoUnitTestProject.AssemblyInfo.cs
[28/04/2023 11:56:12.293 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\obj\Debug\.NETFramework,Version=v4.7.2.AssemblyAttributes.cs
[28/04/2023 11:56:12.294 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\UnitTest2.cs
[28/04/2023 11:56:12.294 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\UnitTest1.cs
[28/04/2023 11:56:12.295 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\Usings.cs
[28/04/2023 11:56:12.295 AM]  Discovering tests in C:\MyDev\Sources\DemoTestBase\obj\Debug\DemoTestBase.AssemblyInfo.cs
[28/04/2023 11:56:12.295 AM]  Discovering tests in C:\MyDev\Sources\DemoTestBase\TestBase.cs
[28/04/2023 11:56:12.296 AM]  Discovering tests in C:\MyDev\Sources\DemoTestBase\obj\Debug\.NETStandard,Version=v2.0.AssemblyAttributes.cs
[28/04/2023 11:56:12.296 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\obj\Debug\net6.0\DemoUnitTestProject.AssemblyInfo.cs
[28/04/2023 11:56:12.296 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\obj\Debug\net6.0\.NETCoreApp,Version=v6.0.AssemblyAttributes.cs
[28/04/2023 11:56:12.296 AM]  Discovering tests in C:\Users\leef109860\.nuget\packages\microsoft.net.test.sdk\17.5.0\build\netcoreapp3.1\Microsoft.NET.Test.Sdk.Program.cs
[28/04/2023 11:56:12.297 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\UnitTest2.cs
[28/04/2023 11:56:12.297 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\UnitTest1.cs
[28/04/2023 11:56:12.297 AM]  Discovering tests in C:\MyDev\Sources\DemoUnitTestProject\Usings.cs

But it seems to be able to discover all tests.

When I try to execute all tests for .Net framework 4.7.2, all passed.

When I try to execute the UnitTest1.TestMethod1 (inherits DemoTestBase.TestBase) for .Net 6.

When I try to execute UnitTest2.TestMethod1 (not inherit DemoTestBase.TestBase) for .Net 6, it failed with message

Test method DemoUnitTestProject.UnitTest2.TestMethod1 threw exception: 
System.IO.FileNotFoundException: Could not load file or assembly 'DemoProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

Unit tests for .Net framework 4.7.2 execute successfully while unit tests for .Net 6 failed with error loading the project reference dependencies, even using the .runsettings file.

Then I tried to execute the test using vstest.console.exe, same result, success for .Net 472 but failed for .Net 6.

Command-line for .Net framework 4.7.2

vstest.console.exe "C:\MyDev\Sources\unittest\DemoUnitTestProject.dll" /Settings:"C:\MyDev\Sources\.runsettings" /Logger:trx;LoggerFileName="C:\MyDev\Sources\trx.log"

Microsoft (R) Test Execution Command Line Tool Version 17.5.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed TestMethod1 [9 ms]
  Passed TestMethod1 [< 1 ms]
Results File: C:\MyDev\Sources\TestResults\leef109860_LT-22-592_2023-04-28_11_34_53.trx

Test Run Successful.
Total tests: 2
     Passed: 2
 Total time: 0.4376 Seconds

Command-line for .Net 6

vstest.console.exe "C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll" /Settings:"C:\MyDev\Sources\.runsettings" 
/Logger:trx;LoggerFileName="C:\MyDev\Sources\trx6.log"

Microsoft (R) Test Execution Command Line Tool Version 17.5.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[MSTest][Discovery][C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll] Unable to load types from the test source 'C:\MyDev\Sources\unittest\net6.0\DemoUnitTestProject.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'DemoTestBase, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

  Failed TestMethod1 [11 ms]
  Error Message:
   Test method DemoUnitTestProject.UnitTest2.TestMethod1 threw exception:
System.IO.FileNotFoundException: Could not load file or assembly 'DemoProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
  Stack Trace:
      at DemoUnitTestProject.UnitTest2.TestMethod1()

Results File: C:\MyDev\Sources\TestResults\leef109860_LT-22-592_2023-04-28_11_35_36.trx

Test Run Failed.
Total tests: 1
     Failed: 1
 Total time: 0.5065 Seconds

I then enabled diagnostic log (/diag:diag6.txt) for the .Net 6 command line, I can see that in the AssemblyResolver.OnResolve, it tries to load the project reference from the unit test project output folder, but not using the Directory path in the .runsettings.

TpTrace Information: 0 : 20916, 4, 2023/04/28, 11:37:38.906, 6677529849663, testhost.dll, AssemblyResolver.OnResolve: DemoTestBase: Resolving assembly.
TpTrace Information: 0 : 20916, 4, 2023/04/28, 11:37:38.906, 6677529850108, testhost.dll, AssemblyResolver.OnResolve: DemoTestBase: Searching in: 'C:\MyDev\Sources\unittest\net6.0'.
TpTrace Information: 0 : 20916, 4, 2023/04/28, 11:37:38.906, 6677529850683, testhost.dll, AssemblyResolver.OnResolve: DemoTestBase: Assembly path does not exist: 'C:\MyDev\Sources\unittest\net6.0\DemoTestBase.dll', returning.
TpTrace Information: 0 : 20916, 4, 2023/04/28, 11:37:38.906, 6677529851064, testhost.dll, AssemblyResolver.OnResolve: DemoTestBase: Assembly path does not exist: 'C:\MyDev\Sources\unittest\net6.0\DemoTestBase.exe', returning.
TpTrace Information: 0 : 20916, 4, 2023/04/28, 11:37:38.906, 6677529851211, testhost.dll, AssemblyResolver.OnResolve: DemoTestBase: Failed to load assembly.

I am not sure why for .Net framework 4.7.2, it is able to load all project references, but not for .Net 6. Must the project references (or all references) be built into the same folder as the unit test output folder for .Net 6?

I’ve created a solution in github to demonstrate the above scenario, I’ve also uploaded the diagnostic file (diag*.txt) from the command line to the repo. https://github.com/eleshoe/DemoUnitTestProject

Thanks (sorry for the long post).

AB#1813780

[eleshoe]

@Evangelink Thanks for the clarification, it is clear now. Thank you very much.

[fx999dev]

To answer more specifically your questions. No, there will not be any “fix” for supporting the AssemblyResolution property in netcore/net versions of MSTest. But we will update the documentation to specify this is not supported AND we will emit a warning if we detect this property in some netcore run.

@Evangelink We have about 1500 dlls of which 30-50 dlls need to be loaded at runtime depending on the project. <AssemblyResolution> via the .runsettings file gave us the possibility to specify a path where all possible runtime components can be found. Surely we can’t specify “Copy Local” on each of our projects, after which around 3 gigabytes of dlls are copied to the binary folder on each build, which would not only be a huge waste of memory, but also a massive performance loss when building the projects.

We are currently still using .NET Framework, but how should we solve the problem when we move to .NET 6.0 in the future?