"No test is available" with /logger:trx but everything works with /logger:console

Please excuse the interesting PowerShell command. It’s based on the command that Azure Pipelines VSTest task builds. I’m trying to track down the difference between an Azure Pipelines build that skips all tests and a build in a different project that actually runs the tests. It comes down to what looks like a VSTest console bug. I can repro locally using the open-source project I’m looking at. If you need it, I can spend the time to boil down the minimal files needed to demonstrate.

If you change /Logger:trx to /Logger:console OR you remove the /TestAdapterPath parameter, tests begin to be discovered and everything works properly.

C:\Users\Joseph\Source\Repos\PropertyChangedAnalyzers [(b740437...)]> &"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "PropertyChangedAnalyzers.Test\bin\Release\net472\PropertyChangedAnalyzers.Test.dll" /Logger:"trx" /TestAdapterPath:"."
Microsoft (R) Test Execution Command Line Tool Version 16.3.0-preview-20190715-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
No test is available in C:\Users\Joseph\Source\Repos\PropertyChangedAnalyzers\PropertyChangedAnalyzers.Test\bin\Release\net472\PropertyChangedAnalyzers.Test.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.      Test run in progress.Results File: C:\Users\Joseph\Source\Repos\PropertyChangedAnalyzers\TestResults\Joseph_TALIESIN_2019-11-06_17_11_12.trx

Diagnostic logs

verbose_diagnostic_log.txt

It looks like the diagnostic log is full of 283 repetitions of vstest.console.exe failing to load System.Runtime.dll. Why would these failures go away if you change /Logger:trx to /Logger:console or if you remove the /TestAdapterPath parameter?

TpTrace Warning: 0 : 22240, 4, 2019/11/06, 17:16:05.318, 3276125106496, vstest.console.exe, TestPluginDiscoverer: Failed to get types from assembly 'NUnit3.TestAdapter, Version=3.15.1.0, Culture=neutral, PublicKeyToken=4cb40d35494691ac'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
TpTrace Warning: 0 : 22240, 4, 2019/11/06, 17:16:05.318, 3276125107286, vstest.console.exe, LoaderExceptions: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginCache.CurrentDomainAssemblyResolve(Object sender, AssemblyResolveEventArgs args)
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyResolver.AssemblyResolverEvent(Object sender, Object eventArgs)
   at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)

[nohwnd]

There is an option to fail the build on no tests https://github.com/microsoft/vstest/issues/2707

[AndirNotes]

I’ve also got into this issue with migrating to 17.5.0 Microsoft.TestPlatform in our Azure DevOps builds and VSTest@2 task. Any workarounds available? @singhsarab? @Evangelink Could you please explain why it’s by design? What that mean?