Logging from WebApplicationFactory can no longer be captured in 2.2.9

Description

We are developing a ASP.NET Core application and for testing we use the ASP.NET Core TestHost (WebApplicationFactory) to facilitate integration testing. To capture the logging we replace the ILoggerFactory with an implementation that uses the TestContext to write the application logging so it can be used for debugging if tests fail.

Since version 2.2.9 logging is no longer reliably captured.

Steps to reproduce

1. Download this example solution: TestOutputTest.zip
2. Run the test
3. Check the TestContext messages, all the logging output for the request is missing. As a test I added a “Hello from Get” message to the controller, which is also not visible.

Expected behavior

All the messages passed to the TestContext for a single test are captured. If MSTest.TestAdapter and MSTest.TestFramework version 2.2.8 are used the output is captured as expected

Actual behavior

Messages are missing, probably because they are in a different async context.

Environment

.NET SDK 6.0.202 Windows 11 Visual Studio 2022 (17.1.4) MsTest 2.2.9

Also tried using the pre-release version 2.2.10-preview-20220414-01, but same issue there.

[vaganovk]

Fixed similar issue by adding TestServer.PreserveExecutionContext = true; before WebApplicationFactory.CreateClient() call.

The root cause seems to be:

1. MSTest collects test output via AsyncLocal-based string builder, see https://github.com/microsoft/testfx/blob/4a1e386b7599b83b82b2a983cf4ab589807acf60/src/Adapter/PlatformServices.Shared/netstandard1.3/Services/ns13ThreadSafeStringWriter.cs#L20 - so when execution context changes, logs get captured into different destination (which then gets ignored after test completes).
2. By default, TestServer does not preserve execution context between client and server calls for a purpose: https://github.com/dotnet/aspnetcore/issues/10134

Capturing test results into separate file and adding it to TestContext seems to be even better option as the intent of PreserveExecutionContext = false was to improve test isolation.

[Evangelink]

I confirm that there is a big difference of output between MSTest < 2.2.9 and above.

Before:

TestContext Messages:
Debug: Microsoft.AspNetCore.Mvc.ModelBinding.ModelBinderFactory [RegisteredModelBinderProviders] Registered model binder providers, in the following order: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider
Debug: Microsoft.Extensions.Hosting.Internal.Host [Starting] Hosting starting
Debug: Microsoft.AspNetCore.Hosting.Diagnostics [HostingStartupAssemblyLoaded] Loaded hosting startup assembly WebApplication1
Debug: Microsoft.Extensions.Hosting.Internal.Host [Started] Hosting started
Information: Microsoft.AspNetCore.Hosting.Diagnostics [1] Request starting HTTP/1.1 GET http://localhost/WeatherForecast - -
Debug: Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware [WildcardDetected] Wildcard detected, all requests with hosts will be allowed.
Trace: Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware [AllHostsAllowed] All hosts are allowed.
Debug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher [CandidatesFound] 1 candidate(s) found for the request path '/WeatherForecast'
Debug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher [CandidateValid] Endpoint 'WebApplication1.Controllers.WeatherForecastController.Get (WebApplication1)' with route pattern 'WeatherForecast' is valid for the request path '/WeatherForecast'
Debug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware [MatchSuccess] Request matched endpoint 'WebApplication1.Controllers.WeatherForecastController.Get (WebApplication1)'
Debug: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware [EndpointMatched] Static files was skipped as the request already matched an endpoint.
Warning: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware [FailedToDeterminePort] Failed to determine the https port for redirect.
Information: Microsoft.AspNetCore.Routing.EndpointMiddleware [ExecutingEndpoint] Executing endpoint 'WebApplication1.Controllers.WeatherForecastController.Get (WebApplication1)'
Information: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [ControllerActionExecuting] Route matched with {action = "Get", controller = "WeatherForecast"}. Executing controller action with signature System.Collections.Generic.IEnumerable`1[WebApplication1.WeatherForecast] Get() on controller WebApplication1.Controllers.WeatherForecastController (WebApplication1).
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [FilterExecutionPlan] Execution plan of authorization filters (in the following order): None
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [FilterExecutionPlan] Execution plan of resource filters (in the following order): None
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [FilterExecutionPlan] Execution plan of action filters (in the following order): Microsoft.AspNetCore.Mvc.ModelBinding.UnsupportedContentTypeFilter (Order: -3000), Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter (Order: -2000)
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [FilterExecutionPlan] Execution plan of exception filters (in the following order): None
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [FilterExecutionPlan] Execution plan of result filters (in the following order): Microsoft.AspNetCore.Mvc.Infrastructure.ClientErrorResultFilter (Order: -2000)
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [ControllerFactoryExecuting] Executing controller factory for controller WebApplication1.Controllers.WeatherForecastController (WebApplication1)
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [ControllerFactoryExecuted] Executed controller factory for controller WebApplication1.Controllers.WeatherForecastController (WebApplication1)
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [BeforeExecutingMethodOnFilter] Action Filter: Before executing OnActionExecuting on filter Microsoft.AspNetCore.Mvc.ModelBinding.UnsupportedContentTypeFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [AfterExecutingMethodOnFilter] Action Filter: After executing OnActionExecuting on filter Microsoft.AspNetCore.Mvc.ModelBinding.UnsupportedContentTypeFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [BeforeExecutingMethodOnFilter] Action Filter: Before executing OnActionExecuting on filter Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [AfterExecutingMethodOnFilter] Action Filter: After executing OnActionExecuting on filter Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter.
Information: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [ActionMethodExecuting] Executing action method WebApplication1.Controllers.WeatherForecastController.Get (WebApplication1) - Validation state: Valid
Information: WebApplication1.Controllers.WeatherForecastController [0] Hello from Get
Information: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [ActionMethodExecuted] Executed action method WebApplication1.Controllers.WeatherForecastController.Get (WebApplication1), returned result Microsoft.AspNetCore.Mvc.ObjectResult in 1.0485ms.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [BeforeExecutingMethodOnFilter] Action Filter: Before executing OnActionExecuted on filter Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [AfterExecutingMethodOnFilter] Action Filter: After executing OnActionExecuted on filter Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [BeforeExecutingMethodOnFilter] Action Filter: Before executing OnActionExecuted on filter Microsoft.AspNetCore.Mvc.ModelBinding.UnsupportedContentTypeFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [AfterExecutingMethodOnFilter] Action Filter: After executing OnActionExecuted on filter Microsoft.AspNetCore.Mvc.ModelBinding.UnsupportedContentTypeFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [BeforeExecutingMethodOnFilter] Result Filter: Before executing OnResultExecuting on filter Microsoft.AspNetCore.Mvc.Infrastructure.ClientErrorResultFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [AfterExecutingMethodOnFilter] Result Filter: After executing OnResultExecuting on filter Microsoft.AspNetCore.Mvc.Infrastructure.ClientErrorResultFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [BeforeExecutingActionResult] Before executing action result Microsoft.AspNetCore.Mvc.ObjectResult.
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector [RegisteredOutputFormatters] List of registered output formatters, in the following order: Microsoft.AspNetCore.Mvc.Formatters.HttpNoContentOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StringOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StreamOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector [NoAcceptForNegotiation] No information found on request to perform content negotiation.
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector [SelectingOutputFormatterWithoutUsingContentTypes] Attempting to select an output formatter without using a content type as no explicit content types were specified for the response.
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector [SelectingFirstCanWriteFormatter] Attempting to select the first formatter in the output formatters list which can write the result.
Debug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector [FormatterSelected] Selected output formatter 'Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter' and content type 'application/json' to write the response.
Information: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor [ObjectResultExecuting] Executing ObjectResult, writing value of type 'WebApplication1.WeatherForecast[]'.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [AfterExecutingActionResult] After executing action result Microsoft.AspNetCore.Mvc.ObjectResult.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [BeforeExecutingMethodOnFilter] Result Filter: Before executing OnResultExecuted on filter Microsoft.AspNetCore.Mvc.Infrastructure.ClientErrorResultFilter.
Trace: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [AfterExecutingMethodOnFilter] Result Filter: After executing OnResultExecuted on filter Microsoft.AspNetCore.Mvc.Infrastructure.ClientErrorResultFilter.
Information: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker [ActionExecuted] Executed action WebApplication1.Controllers.WeatherForecastController.Get (WebApplication1) in 16.3132ms
Information: Microsoft.AspNetCore.Routing.EndpointMiddleware [ExecutedEndpoint] Executed endpoint 'WebApplication1.Controllers.WeatherForecastController.Get (WebApplication1)'
Information: Microsoft.AspNetCore.Hosting.Diagnostics [2] Request finished HTTP/1.1 GET http://localhost/WeatherForecast - - - 200 - application/json;+charset=utf-8 45.8921ms
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopping] Hosting stopping
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopping] Hosting stopping
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopped] Hosting stopped
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopped] Hosting stopped

After:

 TestContext Messages:
Debug: Microsoft.AspNetCore.Mvc.ModelBinding.ModelBinderFactory [RegisteredModelBinderProviders] Registered model binder providers, in the following order: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider
Debug: Microsoft.Extensions.Hosting.Internal.Host [Starting] Hosting starting
Debug: Microsoft.AspNetCore.Hosting.Diagnostics [HostingStartupAssemblyLoaded] Loaded hosting startup assembly WebApplication1
Debug: Microsoft.Extensions.Hosting.Internal.Host [Started] Hosting started
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopping] Hosting stopping
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopped] Hosting stopped
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopping] Hosting stopping
Debug: Microsoft.Extensions.Hosting.Internal.Host [Stopped] Hosting stopped