Erratic 502 Error ConnectionToDownstreamServiceError (When big request)

I’m routing to an X service endpoint. On the service side, I always get the routed Ocelot’s call to the endpoint, and the service returns a 400 HTTP code. Everything is logged.

When Ocelot gets the response from the service sometimes returns 400 but other times also returns 502 (ConnectionToDownstreamServiceError exception). I can ensure that services X gets the call and always returns a 400 HTTP code.

IHttpRequester returned an error, setting pipeline error ConnectionToDownstreamServiceError Message: Error connecting to downstream service, exception

{
  "Routes": [
    {
      "DownstreamPathTemplate": "/{path}",
      "DownstreamScheme": "https",
      "Priority": 5,
      "DownstreamHostAndPorts": [
        {
          "Host": "my-downstream-service",
          "Port": 443
        }
      ],
      "UpstreamHttpMethod": [ "Post" ],
      "UpstreamPathTemplate": "/v1/{path}"
    }
  ],
  "GlobalConfiguration": {
    "BaseUrl": "http://localhost"
  }
}

When returns HTTP code 400 (Expected behavior)

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 POST http://localhost:5020/v1/api/operation/runit text/plain 3306540
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: ocelot pipeline started
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: Upstream url path is /v1/api/operation/runit
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: downstream templates are /{path}
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /{path}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: No authentication needed for /v1/api/operation/runit
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: /{path} route does not require user to be authorized
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: Downstream url is https://my-downstream-service/api/endoint
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: 400 (Bad Request) status code, request uri: https://my-downstream-service/api/endpoint
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: setting http response message
dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: no pipeline errors, setting and returning completed response
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HM8PNITBGL5V:00000002, previousRequestId: no previous request id, message: ocelot pipeline finished
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 POST http://localhost:5020/v1/api/operation/runit text/plain 3306540 - 400 282 application/problem+json;+charset=utf-8 616.7168ms

When returns HTTP code 502 (Erratic behavior)

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 POST http://localhost:5020/v1/api/operation/runit text/plain 3306540
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: ocelot pipeline started
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: Upstream url path is /v1/api/operation/runit
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: downstream templates are /{path}
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /{path}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: No authentication needed for /v1/api/operation/runit
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: /{path} route does not require user to be authorized
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: Downstream url is https://my-downstream-service/api/endpoint
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: IHttpRequester returned an error, setting pipeline error
warn: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: Error Code: ConnectionToDownstreamServiceError Message: Error connecting to downstream service, exception: System.Net.Http.HttpRequestException: Error while copying content to a stream.
       ---> System.IO.IOException: The response ended prematurely.
         at System.Net.Http.HttpConnection.FillAsync(Boolean async)
         at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
         at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at Ocelot.Requester.HttpClientHttpRequester.GetResponse(HttpContext httpContext) errors found in ResponderMiddleware. Setting error response for request path:/v1/api/operation/runit, request method: POST
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HM8PNJR0N9BG:00000002, previousRequestId: no previous request id, message: ocelot pipeline finished
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 POST http://localhost:5020/v1/api/operation/runit text/plain 3306540 - 502 0 - 824.4424ms

Specifications

• Version: 16.0.1
• Platform: .NET Core 3.1

[henrysmith]

Chunked transfer encoding is an HTTP/1.1 feature that enables keep-alive requests. To avoid it, until we find a final solution, we can configure the routes to use HTTP/1.0, understanding that it requires a new connection for each resource downloaded.

    {
      "DownstreamPathTemplate": "/{path}",
      "DownstreamScheme": "https",
      "Priority": 5,
      "DownstreamHostAndPorts": [
        {
          "Host": "my-downstream-service",
          "Port": 443
        }
      ],
      "DownstreamHttpVersion": "1.0",
      "UpstreamPathTemplate": "/my/{path}"
    }

[raman-m]

@henrysmith Hi Henry! Thanks for reporting the issue!

Have you find some workaround and/or solution for this bug?

What about to test once again for last package v19.0.2, a release for .NET 7? Will you be able to upgrade your gateway solution to .NET 7 with referencing to 19.0.2 package?