Considerable overhead in middleware pipeline

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

NewRelic monitoring showed that a considerable amount of time is spent in Middleware Pipeline

Traces consistently show a delay between request starting and request arriving at the controller action(measure-http-metrics is custom middleware that only wraps request delegate with a stopwatch for tracing, it’s placed after routing)

We removed from the pipeline everything except UseRouting and UseEndpoints, after turning on Debug logs we can see that controller action execution takes around 8 ms from an overall 21 ms, which doesn’t seem ok.

A request is very small JSON with only 2 properties, the response is relatively big JSON - around 5 KB.

Service is hosted in k8s, uses alpine image

[2022-10-01 13:29:59.589 PM] info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://services.local/api/verification/getstatus application/json 68 [2022-10-01 13:29:59.590 PM] dbug: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[1] POST requests are not supported [2022-10-01 13:29:59.590 PM] dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001] 1 candidate(s) found for the request path ‘/api/verification/getstatus’ [2022-10-01 13:29:59.590 PM] dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005] Endpoint ‘Services.Controllers.VerificationController.GetStatus (Services)’ with route pattern ‘api/Verification/GetStatus’ is valid for the request path ‘/api/verification/getstatus’ [2022-10-01 13:29:59.590 PM] dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1] Request matched endpoint ‘Services.Controllers.VerificationController.GetStatus (Services)’ [2022-10-01 13:29:59.590 PM] info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint ‘Services.Controllers.VerificationController.GetStatus (Services)’ [2022-10-01 13:29:59.591 PM] info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3] Route matched with {action = “GetStatus”, controller = “Verification”}. Executing controller action with signature System.Threading.Tasks.Task`1[Services.Contracts.Responses.Verification.VerificationApiResponse] GetStatus(Services.Contracts.Requests.Verification.GetStatusApiRequest) on controller Services.Controllers.VerificationController (Services). [2022-10-01 13:29:59.591 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1] Execution plan of authorization filters (in the following order): None [2022-10-01 13:29:59.591 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1] Execution plan of resource filters (in the following order): Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.SaveTempDataFilter [2022-10-01 13:29:59.591 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1] Execution plan of action filters (in the following order): Microsoft.AspNetCore.Mvc.ModelBinding.UnsupportedContentTypeFilter (Order: -3000), Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter (Order: -2000) [2022-10-01 13:29:59.591 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1] Execution plan of exception filters (in the following order): None [2022-10-01 13:29:59.591 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1] Execution plan of result filters (in the following order): Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.SaveTempDataFilter, Microsoft.AspNetCore.Mvc.Infrastructure.ClientErrorResultFilter (Order: -2000) [2022-10-01 13:29:59.594 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1] Executing controller factory for controller Services.Controllers.VerificationController (Services) [2022-10-01 13:29:59.594 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2] Executed controller factory for controller Services.Controllers.VerificationController (Services) [2022-10-01 13:29:59.594 PM] dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder[22] Attempting to bind parameter ‘request’ of type ‘Services.Contracts.Requests.Verification.GetStatusApiRequest’ … [2022-10-01 13:29:59.595 PM] dbug: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder[44] Attempting to bind parameter ‘request’ of type ‘Services.Contracts.Requests.Verification.GetStatusApiRequest’ using the name ‘’ in request data … [2022-10-01 13:29:59.595 PM] dbug: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder[1] Selected input formatter ‘Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter’ for content type ‘application/json’. [2022-10-01 13:29:59.595 PM] dbug: Microsoft.AspNetCore.Server.Kestrel[25] Connection id “0HML3M35S3U4L”, Request id “0HML3M35S3U4L:00000205”: started reading request body. [2022-10-01 13:29:59.595 PM] dbug: Microsoft.AspNetCore.Server.Kestrel[26] Connection id “0HML3M35S3U4L”, Request id “0HML3M35S3U4L:00000205”: done reading request body. [2022-10-01 13:29:59.596 PM] dbug: Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter[2] JSON input formatter succeeded, deserializing to type ‘Services.Contracts.Requests.Verification.GetStatusApiRequest’ [2022-10-01 13:29:59.596 PM] dbug: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder[45] Done attempting to bind parameter ‘request’ of type ‘Services.Contracts.Requests.Verification.GetStatusApiRequest’. [2022-10-01 13:29:59.596 PM] dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder[23] Done attempting to bind parameter ‘request’ of type ‘Services.Contracts.Requests.Verification.GetStatusApiRequest’. [2022-10-01 13:29:59.596 PM] dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder[26] Attempting to validate the bound parameter ‘request’ of type ‘Services.Contracts.Requests.Verification.GetStatusApiRequest’ … [2022-10-01 13:29:59.597 PM] dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder[27] Done attempting to validate the bound parameter ‘request’ of type ‘Services.Contracts.Requests.Verification.GetStatusApiRequest’. [2022-10-01 13:29:59.598 PM] info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1] Executing action method Services.Controllers.VerificationController.GetStatus (Services) - Validation state: Valid [2022-10-01 13:29:59.606 PM] info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2] Executed action method Services.Controllers.VerificationController.GetStatus (Services), returned result Microsoft.AspNetCore.Mvc.ObjectResult in 7.9254ms. [2022-10-01 13:29:59.607 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[11] List of registered output formatters, in the following order: Microsoft.AspNetCore.Mvc.Formatters.HttpNoContentOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StringOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StreamOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter [2022-10-01 13:29:59.607 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[4] No information found on request to perform content negotiation. [2022-10-01 13:29:59.608 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[8] Attempting to select an output formatter without using a content type as no explicit content types were specified for the response. [2022-10-01 13:29:59.608 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[10] Attempting to select the first formatter in the output formatters list which can write the result. [2022-10-01 13:29:59.608 PM] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[2] Selected output formatter ‘Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter’ and content type ‘application/json’ to write the response. [2022-10-01 13:29:59.608 PM] info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1] Executing ObjectResult, writing value of type ‘Services.Contracts.Responses.Verification.VerificationApiResponse’. [2022-10-01 13:29:59.609 PM] info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2] Executed action Services.Controllers.VerificationController.GetStatus (Services) in 17.3468ms [2022-10-01 13:29:59.610 PM] info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint ‘Services.Controllers.VerificationController.GetStatus (Services)’ [2022-10-01 13:29:59.610 PM] info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://services.local/api/verification/getstatus application/json 68 - 200 - application/json;+charset=utf-8 21.4094ms

Expected Behavior

What can be possible reasons for such behavior? Can somebody point me in the right direction? From logs, it doesn’t seem that problem is in network/serialization.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

6.0.302

Anything else?

~ $ dotnet --info

global.json file: Not found

Host: Version: 6.0.7 Architecture: x64 Commit: 0ec02c8c96

.NET SDKs installed: No SDKs were found.

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Download .NET: https://aka.ms/dotnet-download

Learn about .NET Runtimes and SDKs: https://aka.ms/dotnet/runtimes-sdk-info

[avandych]

@adityamandaleeka validated, after disabling nagle’s algorithm on the client applications that produced majority of traffic, overhead in the middleware pipeline dropped under 1ms. Seems like our initial approach for reproducing the behavior wasn’t correct from the start and the overhead that we received while testing was really from verbose logging, which created confusion. Thanks for your help!

[msftbot[bot]]

Hi @avandych. We have added the “Needs: Author Feedback” label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.