Unable to run unit tests remotely on Test Agent using MSTest V2

Description

I am trying to run my unit tests with UI automation on a Visual Studio Test Controller and Visual Studio Test Agent but I get the following warning Warning : A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter. and none of my tests load.

Steps to reproduce

• Add a .testsettings file to your solution
• Add a .runsettings file to your solution
• Add a SettingsFile element to the MSTest element in the .runsettings
• Add a ForceLegacyMode element to the MSTest element in the .runsettings
• Select the .runsettings as the active test settings (Test -> Test Settings -> Select Test Settings File)
• Rebuild Solution

Expected behavior

Tests are loaded and I am able to run my unit tests on the Test Controller. I understand the legacy mode not being supported but it seems like the .runsettings should also have a means for me to specify the host for the test controller.

Actual behavior

Tests are not loaded and I get the following warnings. When I remove loading the SettingsFile and the ForceLegacyMode elements my tests load but then I won’t be able to connect to the controller.

[11/22/2017 9:54:12 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:13 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:13 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:14 AM Warning] Warning : A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter.

Environment

Just to clarify I am not doing coded UI tests, I am running unit tests that are using Microsoft’s UIA through a open source project called FlaUI. I am trying to use the test controller as part of our CD CI pipeline to run out UI automation on producing builds.

Here are my .runsettings and .testsettings

.runsettings

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Configurations that affect the Test Framework -->
  <RunConfiguration>
    <MaxCpuCount>1</MaxCpuCount>
    <!-- Path relative to solution directory -->
    <ResultsDirectory>.\TestResults</ResultsDirectory>

    <!-- [x86] | x64    
      - You can also change it from menu Test, Test Settings, Default Processor Architecture -->
    <TargetPlatform>x86</TargetPlatform>

    <!-- Framework35 | [Framework40] | Framework45 -->
    <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
  </RunConfiguration>

  <!-- Configurations for data collectors -->
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
              </Exclude>
            </ModulePaths>

            <!-- We recommend you do not change the following values: -->
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>

          </CodeCoverage>
        </Configuration>
      </DataCollector>

    </DataCollectors>
  </DataCollectionRunSettings>

  <!-- Parameters used by tests at runtime -->
  <TestRunParameters>
  </TestRunParameters>

  <!-- Adapter Specific sections -->

  <!-- MSTest adapter -->
  <MSTest>
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <CaptureTraceOutput>false</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>False</DeploymentEnabled>
    <SettingsFile>test.testsettings</SettingsFile>
    <ForcedLegacyMode>true</ForcedLegacyMode>
    <AssemblyResolution>
      <Directory Path=".\Tests" includeSubDirectories="true"/>
    </AssemblyResolution>
  </MSTest>

</RunSettings>

.testsettings

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="rmoqa01" id="076be28c-d18b-46bf-ad20-4d43ec821ea4" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <RemoteController name="10.2.0.101" />
  <Execution location="Remote">
    <Hosts skipUnhostableTests="false">
      <VSSDKTestHostRunConfig name="VS IDE" HiveKind="DevEnv" HiveName="15.0_c9b36733" xmlns="http://microsoft.com/schemas/VisualStudio/SDK/Tools/IdeHostAdapter/2006/06" />
    </Hosts>
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution>
          <TestDirectory useLoadContext="true" />
        </AssemblyResolution>
      </UnitTestRunConfig>
    </TestTypeSpecific>
    <AgentRule name="AllAgentsDefaultRole">
    </AgentRule>
  </Execution>
  <Properties />
</TestSettings>

[VanKrock]

How I can downgrade to MSTest v1?

[Tyale-cpu]

I have to say that is terrible news for us. The assumption there is that we are using VSTS as part of our CI process, which we don’t. We use TeamCity to build our products and Jenkins to execute our tests and I can’t believe we are the only ones not using VSTS. It really is a shame as we had high hopes for the new framework.

I do appreciate the follow up, just disappointed.

Tom