[Epic] Request Throttling and Queuing Middleware

Optimistically putting this in 3.1, but no committments as to when exactly this ships. It won’t be part of 3.0.

This is tracking work to add a Request Throttling and Queuing middleware (or possibly middlewares) to ASP.NET Core. The purpose of this is to enable app developers to restrict the number of concurrently-running requests in their application in order to better manage resources.

Currently, the only queuing that occurs in ASP.NET Core applications is through use of the thread-pool queue, which queues completions of async operations (i.e. async/await) and other thread-pool work items. This can be difficult to manage and measure. For some applications, the introduction of a concurrent request limit can mitigate issues caused by problematic async patterns (like “sync-over-async” where blocking calls are done in async code paths). The “Right Thing” to do in most of these cases is to unravel these “sync-over-async” calls into a true async/await pattern, but sometimes that’s infeasible or too costly. Limiting concurrent requests conserves these resources.

When there is a concurrent request limit, it is also desirable to “queue” requests that come in over this limit rather than turning them away immediately. This also provides a clear metric to use to measure the impact of load on your system (the number of requests waiting in the queue).

Our goal is to build a middleware that limits the number of concurrently executing requests in the remainder of the pipeline. The middleware will also queue requests beyond that limit until there is space. Finally, if the limit and the queue are exhausted, it will reject the request immediately (usually by emitting a 503 HTTP status code).

We do believe that components of this middleware may be useful for “throttling” as well. Throttling includes scenarios like API Rate Limits, where requests are tracked against certain per-user limits and rejected when they exceed that limit. This is beyond the current scope of the middleware but may serve as a stretch goal.

NOTE: This is our summer intern project, so bear with us as we work through it. We’ve shipped many of our intern projects as production code in future releases and intend to do so with this one as well!

Child Items:

• Basic middleware structure.
• Concurrent request limit.
• Queuing requests beyond the limit.
• https://github.com/aspnet/AspNetCore/issues/10703 Microbenchmarks.
• https://github.com/aspnet/AspNetCore/issues/10704 Measure the overhead of queuing in the Plaintext benchmark.
• https://github.com/aspnet/AspNetCore/issues/10705 Add Event Counters and use dotnet-counter tool to view them.
• https://github.com/aspnet/AspNetCore/issues/11339 Add benchmark scenarios to illustrate where Request Queuing benefits
• https://github.com/aspnet/AspNetCore/issues/10706 Optimizations based on benchmarks. Use a profiler to explore further.
• https://github.com/aspnet/AspNetCore/issues/10707 OnRejected callback to allow user code to handle requests rejected due to insufficient queue space.
• Multiple queues and queue assignment. (Func<HttpContext, Queue>, 💪 stretch goal)

[analogrelay]

Our plans (for now) are limited to what you see above 😃. Simple front-door queueing is what we’re starting with, we do have some ideas around integrating throttling moving forward but that may also end up as a separate middleware.

[halter73]

So for example, authenticated user traffic could be prioritized?

If we reach our stretch goal and get multi-queue support, that should be possible.