MSTestAdapter failed to discover tests because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' does not have an implementation

Description

We’ve long had a running VSTest pipeline for our projects unit tests, which target Framework 4.8 and run MSTest.

The YAML of the unit test on DevOps is very minimal:

- name: DebugBuildFolder
  default: '$(DebugBuildFolder)'
- name: DebugBuildArtifactName
  default: '$(DebugBuildArtifactName)'

steps:
  - task: DownloadPipelineArtifact@2
    displayName: 'Download ${{ parameters.DebugBuildArtifactName }} artifact'
    inputs:
      buildType: 'current'
      artifactName: ${{ parameters.DebugBuildArtifactName }}
      targetPath: '$(Build.SourcesDirectory)'
  - task: VSTest@2
    displayName: 'UnitTests on debug build'
    inputs:
      testAssemblyVer2: '${{ parameters.DebugBuildFolder }}\FrameworkUnitTests.dll'

We recently added an empty MSTest project to our solution, which targets Core 6 instead:

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>

    <IsPackable>false</IsPackable>

    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.8" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.8" />
    <PackageReference Include="coverlet.collector" Version="3.1.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

</Project>

With that project simply present, an error message appears and not a single unit test is discovered. Despite that new MSTest project not being referenced by any other project, and despite VSTest correctly only being set on the same FrameworkUnitTests.dll as before. For every test class xxxTests, the log contains a message of the form:

[MSTest][Discovery][D:\a\1\s\Build\Debug\UnitTests.dll] MSTestAdapter failed to discover tests in class FrameworkUnitTests.xxxTests' of assembly 'D:\a\1\s\Build\Debug\FrameworkUnitTests.dll' because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' from assembly 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' does not have an implementation..

and at the end there is a message:

No test is available in D:\a\1\s\Build\Debug\FrameworkUnitTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

The problem does not arise in Visual Studio, I guess because it doesn’t need the TestAdapter but runs the tests directly. Notably, no project in the solution referenced the new MSTest project or it’s single (empty, default) unit test. As soon as we remove the new Core MSTest project from the solution, all tests get discovered again. Weird!

Now, I was able to discover the potential root of the problem:

The Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll in the build output folder has a different size when the Core MSTest project is present - it’s just 59kb, compared to109kb before. I then discovered that because of the Core 6 project, the NuGet reference to MSTest.TestAdapter - which both shows as version 2.2.8 - will actually resolve to different dlls than before. Namely, it uses the dll from mstest.testadapter\2.2.8\build\netstandard1.5 instead of mstest.testadapter\2.2.8\build_common, which targets Framework 4.5

The class TestContextImplementation in Framework 4.5 contains a public DataRow attribute, while the TestContextImplementation in Standard 1.5 does not. The moment the Core project is added to the solution, MSBuild will copy the Standard 1.5 dll into the output directory. But the Framework 4.5 UnitTest.dll was linked against the Framework 4.5 version, where the DataRow attribute IS present. When VSTest loads that library it discovers that the implementation for DataRow is missing and quits. (I don’t know why Standard 1.5 doesn’t need this implementation, since it also should have to implement https://docs.microsoft.com/de-de/dotnet/api/microsoft.visualstudio.testtools.unittesting.testcontext?view=visualstudiosdk-2022 - but it apparently doesn’t?)

I don’t currently know how to solve this problem - there seems to be no way to force the 4.8 project to also use the Standard 1.5 version for the MSTest package, thereby linking against the same library that the other project uses, see also: https://github.com/dotnet/sdk/issues/1791 https://github.com/NuGet/Home/issues/7416

I don’t know if it is actually well-defined which Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll gets copied to the output folder of a build, is there a way to control this? After all there is a factual conflict between the two libraries - but maybe this isn’t discovered since both officially register as MSTest 2.2.8. We consistently see the Standard 1.5 library copied to the output on our Azure DevOps pipeline, so the problem means no tests are discovered for our Framework VSTest above.

One workaround I see right now would be to put the unit tests for Core and Framework into different solutions. It would make sense for us to use the same solution however, since we have projects targetting Framework, newer ones targetting Core and some common projects referenced by both the Framework and Core projects targetting Standard 2.0 for compatibility. In any case, MSTest surely shouldn’t just break when both Framework and Core projects use it in a solution.

Another workaround might be be to remove the TestAdapter libraries from the build output folder and run VSTest with a /TestAdapterPath parameter pointing to the correct MSTest version, depending on which project gets tested. But that’s rather hacky, especially since I wouldn’t know of a nice way to always copy one version of the NuGet package to, say, a folder “FrameworkTestAdapter” and the other to “CoreTestAdapter” so I can reference these in different VSTest tasks.

But maybe there is a way to fix this problem “properly”? For instance, I don’t see why it’s necessary that the TestContextImplementation exposes public members that simply aren’t present in the

Steps to reproduce

Create and build two MSTest projects in the same solution, one targetting Core, the other targetting Framework 4.8

Expected behavior

The Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll that gets copied to the build folder works for running both MSTest projects

Actual behavior

The above dll targeting netstandard1.5 is copied to the build folder. For the Framework MSTest project, running the unit tests from the dll via vstest.console will produce an error: [MSTest][Discovery][D:\a\1\s\Build\Debug\UnitTests.dll] MSTestAdapter failed to discover tests in class FrameworkUnitTests.xxxTests' of assembly 'D:\a\1\s\Build\Debug\FrameworkUnitTests.dll' because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' from assembly 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' does not have an implementation..

Environment

• Operating system: Windows 11
• Build version of vstest.console: 17.0.0
• Package version of MSTest framework and adapter: 2.2.8
• Other installed packages and their versions on the test project: see above

[HarmanveerKaur]

@Evangelink I have sent the details on your microsoft email. Kindly take a look.

[wangyuan8421]

Ah, @Evangelink, I checked the log and it missing get_properties. I will create a new issue then