Getting test output that breaks Visual Studio Devops Build pipeline (MSTest.Adapter v2.2.3)

Description

I was holding back on v2.2.1 and using v2.1.2 because of issue #789

When I upgraded to v2.2.2 and then to v2.2.3 I find the tests are running inside Visual Studio and the breaking behavior in 2.2.1 is resolved; however after running all the tests In the Test Output window I notice the following message

---------- Starting test run ----------
MSTest Executor: Test Parallelization enabled for C:\Path\To\My\Test\Project\bin\x64\Debug\net48\Tests.Integration.dll (Workers: 12, Scope: ClassLevel).
An exception occurred while invoking executor 'executor://mstestadapter/v2': The parameter cannot be null or empty.
Parameter name: message
========== Test run finished: 693 Tests run in 48.9 sec (693 Passed, 0 Failed, 0 Skipped) ==========

This doesn’t cause any issues in Visual studio but when they are executed in Visual Studio DevOps pipeline, the mention of the word exception causes the Pipeline task to fail, even though all tests have passed.

I have rolled back to 2.1.2 multiple times, and after reverting, the above error message goes away.

Steps to reproduce

1. Take test project with MSTest.Adapter version 2.1.2
2. Upgrade to 2.2.3

Expected behavior

All tests should pass without any messages mentioning any exception, e.g. when running same tests with v2.1.2 give the following message at end of run

---------- Starting test run ----------
MSTest Executor: Test Parallelization enabled for C:\Path\To\My\Test\Project\bin\x64\Debug\net48\Tests.Integration.dll (Workers: 12, Scope: ClassLevel).
========== Test run finished: 693 Tests run in 50.4 sec (693 Passed, 0 Failed, 0 Skipped) ==========

Actual behavior

I get the following message

---------- Starting test run ----------
MSTest Executor: Test Parallelization enabled for C:\Path\To\My\Test\Project\bin\x64\Debug\net48\Tests.Integration.dll (Workers: 12, Scope: ClassLevel).
An exception occurred while invoking executor 'executor://mstestadapter/v2': The parameter cannot be null or empty.
Parameter name: message
========== Test run finished: 693 Tests run in 48.9 sec (693 Passed, 0 Failed, 0 Skipped) ==========

I could be missing some Run parameters, but I couldn’t figure out which one exactly, quickly. Any pointers appreciated.

Environment

Visual Studio 2019 16.9.1 Windows 10 Enterprise x64 Test environment set to x64 explicitly Nuget Package Style: PackageReference

Package references in the project

<ItemGroup>
    <PackageReference Include="Castle.Core" Version="4.4.1" />
    <PackageReference Include="EfCore.TestSupport" Version="3.2.0" />
    <PackageReference Include="FluentAssertions" Version="5.10.3" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" Version="3.1.13" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite.NetTopologySuite" Version="3.1.13" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer.NetTopologySuite" Version="3.1.13" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="5.0.0" />
    <PackageReference Include="Moq" Version="4.16.1" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.3" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.3" />
    <PackageReference Include="Newtonsoft.Json.Schema" Version="3.0.14" />
    <PackageReference Include="sqlite" Version="3.13.0" />
    <PackageReference Include="System.ValueTuple" Version="4.5.0" />
  </ItemGroup>

[odalet]

Well, to be honest, it was sheer luck that lead me to this workaround: I was merely trying to dump some information while investigating with my Console.WriteLine stuff… And then, I’m not that surprised the hack works with assembly level init/cleanup code as well: ultimately it taps into the same LogCleanupResult method deep down the test adapter.

@Haplois I think the parts to investigate are what happens during initialization and cleanup steps relating to console redirections. Mongo2go may be specifically at fault (although it didn’t bother MSTest v2.1), but I suspect we’d end up with the same errors if we were to use any code that spawns a process and monitors its output.

I know from experience that Windows Console redirection is a b*tch to get right so I had a look at how this behaves on Linux: I ran my test test on WSL and… it just works!

[Haplois]

Hi @odalet - thank you for all your work here. Since you did all the investigation, if you open the PR; I can guide you on the tests.