TestContext.Progress shows up as Warning in TFS/Azure Dev Ops

Version information

• NUnit3TestAdapter version: 4.3.1.0
• NUnit 3.13.3
• Visual Studio edition: Microsoft Visual Studio Professional 2022 (64-bit) - Current Version 17.4.2
• What .net platform and version is being targeted: .Net 7.0
• TFS/VSTS: Azure DevOps Server Version Dev18.M170.8 on premise
  • Task: VSTest@2 (https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines&viewFallbackFrom=azure-devops)

Issue

TestContext.Progress written during a test shows up in TFS/Azure DevOps as warning when running a test via a VSTest task with NUnit Adapter 4.3.1.0. ConsoleOut is enabled via runsettings.

Code used to write progress in test: TestContext.Progress.WriteLine("Message via TestContext.Progress"); Expected result white (informational) log line in TFS. Instead the message is shows as a warning (orange) warning in TFS:

Possible cause

My theory regarding the cause of the issue based on analysis of the code (did not run/debug the code): The text written via TestContext.Progress.WriteLine results in a TestOutput event in EventListenerTextWriter:

private bool TrySendToListener(string text) { context.Listener.TestOutput(new TestOutput(text, _streamName, context.CurrentTest?.Id, context.CurrentTest?.FullName));

The TestOutput event is in handled in nunit3-vs-testadapter in class NUnitEventListener OnTestEvent:

public void OnTestEvent(string report) { . . . case NUnitTestEventHeader.EventType.TestOutput: TestOutput(new NUnitTestEventTestOutput(node)); break;

The handled TestOutput is always forwarded as warning in NUnitEventListener TestOutput:

public void TestOutput(INUnitTestEventTestOutput outputNodeEvent) { . . . recorder.SendMessage(TestMessageLevel.Warning, text); }

Possible solution: recorder.SendMessage(outputNodeEvent.IsProgressStream ? TestMessageLevel.Informational : TestMessageLevel.Warning, text); or recorder.SendMessage(outputNodeEvent.IsErrorStream ? TestMessageLevel.Warning : TestMessageLevel.Informational, text); Further more one could argue that if settings.ConsoleOut == 1 the message level should also be TestMessageLevel.Informational, similar to the message sent in NUnitEventListener TestFinished.

Apologies if the issue turns out to be caused by something else entirely (e.g. configuration on my side).

[mharwig]

@OsirisTerje Hi, Thanks for the update.

I just updated to Version 4.4.0-beta.1 in the reproducer, and it works as intended.

Run with version 4.3.1: https://martinharwig.visualstudio.com/NUnitVsTestAdapterBug/_build/results?buildId=14&view=results

Run with Version 4.4.0-beta.1 (no more warning with progress message): https://martinharwig.visualstudio.com/NUnitVsTestAdapterBug/_build/results?buildId=15&view=results

[mharwig]

@OsirisTerje I realize that those settings are not related to the adapter. However using customBatchSizes has the side effect that TFS runs the tests via the translation layer (see comment here: https://github.com/microsoft/azure-pipelines-tasks/issues/10709#issuecomment-508416719), as opposed to calling vstest.console.exe directly. Apparently the variant ran via the api handles the warning messages differently. I will see if I can find the time to create a reproducer.