vstest.console.exe returns a zero exit code even if no tests were found/executed

Description

vstest.console.exe returns a zero exit code even if no tests were found/executed. A zero exit code usually means success, but since no tests were found/executed, I expected a non-zero exit code meaning a failure occurred.

Steps to reproduce

Create and build a new Unit Test Project (.NET Framework 4.6.1) in Visual Studio 2017 v15.9.17

Create a `Local.testsettings’ file:

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local" id="068518c9-0930-453f-bf19-5e97cdf5545e" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
  <Execution>
    <Timeouts runTimeout="9000000" testTimeout="9000000" />
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution>
          <TestDirectory useLoadContext="true" />
        </AssemblyResolution>
      </UnitTestRunConfig>
      <WebTestRunConfiguration testTypeId="4e7599fa-5ecb-43e9-a887-cd63cf72d207">
        <Browser name="Internet Explorer 9.0" MaxConnections="6">
          <Headers>
            <Header name="User-Agent" value="Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" />
            <Header name="Accept" value="*/*" />
            <Header name="Accept-Language" value="{{$IEAcceptLanguage}}" />
            <Header name="Accept-Encoding" value="GZIP" />
          </Headers>
        </Browser>
      </WebTestRunConfiguration>
    </TestTypeSpecific>
    <AgentRule name="LocalMachineDefaultRole">
    </AgentRule>
  </Execution>
  <Properties />
</TestSettings>

Run "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "c:\unittests\MyUnitTestProject\MyUnitTestProject.dll" /tests:MyUnitTestProject.UnitTest1.TestMethod1 /logger:trx /settings:"c:\unittests\Local.testsettings"

Another way to reproduce the issue is to just pass a test method that doesn’t exist.

Ex., run "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "c:\unittests\MyUnitTestProject\MyUnitTestProject.dll" /tests:MyUnitTestProject.UnitTest1.TestMethod2

Expected behavior

I expected a non-zero exit code

Actual behavior

vstest.console.exe exited with a zero exit code.

Console Output:

Microsoft (R) Test Execution Command Line Tool Version 15.9.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test discovery, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.5 and platform X86. Following DLL(s) do not match framework/platform settings.
MyUnitTestProject.dll is built for Framework 4.6.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.

TestSettings will soon be deprecated for automated unit and functional testing scenarios. It is recommended that you use RunSettings. To learn more, see http://aka.ms/runsettings
Warning : A testsettings file, a runsettings file with a ForcedLegacyMode set to true or a vsmdi file is not supported with the MSTest V2 Adapter.
No test is available in c:\unittests\MyUnitTestProject\MyUnitTestProject.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
No test is available in c:\unittests\MyUnitTestProject\MyUnitTestProject.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again. Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

In the case where the test method doesn’t exist:

Microsoft (R) Test Execution Command Line Tool Version 15.9.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test discovery, please wait...
A total of 1 tests were discovered but no test matches the specified selection criteria(MyUnitTestProject.UnitTest1.TestMethod2). Use right value(s) and try again.

Environment

Windows Server 2016 64-bit Visual Studio Test Agent 2017 15.9.28307.858 MSTest.TestAdapter v1.3.2 MSTest.TestFramework v1.3.2

[Volkmire]

@singhsarab Please re-open the issue. We constantly encounter it during automated pipeline setups. A couple of examples:

• no test.dlls published -> no tests run -> visible result is success;
• incorrect folder specified during publishing/execution -> no tests run -> visible result is success;
• failure to download test.dlls -> no tests run -> visible result is success;

In all of those cases we expect the runner to fail, because it had 1 job, to run tests and didn’t do it. If direct implementation would be breaking, considering adding a custom flag to toggle this behavior.

[pavelhorak]

Expected behaviour is that vstest.console.exe command, like any other external command, exits with status code 0 if it terminates normally. This also applies to a situation where no tests meet the selection criteria. Now, we won’t change this as a DEFAULT behaviour, however, to satisfy the needs, we will introduce an option (new .runsettings parameter and command line switch like suggested by @icnocop) to override this behaviour and exit vstest.console.exe with non zero code if no test was executed.