Microsoft.AspNetCore.Mvc.Testing : The request was aborted or the pipeline has finished on TeamCity

Describe the bug

Our Integration Suite on TeamCity keeps on failing randomly but runs fine locally from Visual Studio and command line as well.

To Reproduce

Steps to reproduce the behavior:

1. Using this version of ASP.NET Core ‘2.1.1’
2. Run tests on TeamCity

Expected behavior

Assert errors but no errors from TestHost

Screenshots

Stack trace from TeamCity:

 Failed   Agoda.Ycs.Gateway.IntegrationTests.GetCakesEndpointTest.CakesEndpointTest(url: "/api/Cakes/1")
 Error Message:
  System.IO.IOException : The request was aborted or the pipeline has finished
 Stack Trace:
    at Microsoft.AspNetCore.TestHost.ResponseStream.CheckNotComplete()
    at Microsoft.AspNetCore.TestHost.ResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.FlushInternalAsync(Boolean flushEncoder)
    at Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultAsync(IActionResult result)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResultFilterAsync[TFilter,TFilterAsync]()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResultExecutedContext context)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultFilters()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
    at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
    at Agoda.Ycs.Gateway.Api.Middlewares.MeasurementsLoggingMiddleware.Invoke(HttpContext context, IOptionsSnapshot`1 adpMessagingClientSettings)
    at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
    at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)
    at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)
    at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
    at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
    at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
    at Microsoft.AspNetCore.TestHost.HttpContextBuilder.<>c__DisplayClass10_0.<<SendAsync>b__0>d.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
    at Microsoft.AspNetCore.TestHost.ClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.Mvc.Testing.Handlers.CookieContainerHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.Mvc.Testing.Handlers.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
    at Agoda.Ycs.Gateway.IntegrationTests.GetEndpointTestBase`1.EndpointsReturnSuccessAndCorrectContentType(String url) in /opt/buildagent/work/69f9ec429751afcd/tests/Agoda.Ycs.Gateway.IntegrationTests/GetEndpointTestBase.cs:line 16
    at Agoda.Ycs.Gateway.IntegrationTests.GetCakesEndpointTest.CakesEndpointTest(String url) in /opt/buildagent/work/69f9ec429751afcd/tests/Agoda.Ycs.Gateway.IntegrationTests/GetCakesEndpointTest.cs:line 15

Additional context

• Teamcity agents are running Linux, version 3.10.0-862.3.3.el7.x86_64

[csMACnz]

I found this issue from similar symptoms and wanted to add an extra bit of information that helped me diagnose, for the next person who sees this.

Something in the request was taking too long, which was triggering the HttpClient’s default timeout of 100 seconds. AspNetCore Cancellation Tokens don’t cause an immediate failure, but instead will trigger when you try to write your response, giving the error above “The request was aborted or the pipeline has finished” at that point.

Adding internal timeouts to the internal Http calls inside your service helps to find the cause of the slow requests, and trigger the failure where it is slow rather than after it is slow.

Otherwise we basically had the same issue as @Ibrahim-Islam (slow upstream responses)

[artisticcheese]

Increasing default HTTPClient timeout for me fixed it. client.Timeout = System.TimeSpan.FromMinutes(3);