ASP.NET Core 2.2 kestrel server's performance issue

※Describe the bug

I’m facing problem with kestrel server’s performance. I have following scenario :

TestClient(JMeter) -> DemoAPI-1(Kestrel) -> DemoAPI-2(IIS) 

I’m trying to create a sample application that could get the file content as and when requested. TestClient(100 Threads) requests to DemoAPI-1 which in turn request to DemoAPI-2. DemoAPI-2 reads a fixed XML file(1 MB max) and returns it’s content as a response(In production DemoAPI-2 is not going to be exposed to outside world).

When I tested direct access from TestClient -> DemoAPI-2 I got expected result(good) which is following :

• Average : 368ms
• Minimum : 40ms
• Maximum : 1056ms
• Throughput : 40.1/sec

But when I tried to access it through DemoAPI-1 I got following result :

• Average : 48232ms
• Minimum : 21095ms
• Maximum : 49377ms
• Throughput : 2.0/sec

As you can see there is a huge difference.I’m not getting even the 10% throughput of DemoAPI-2. I was told kestrel is more efficient and fast compared to traditional IIS. Also because there is no problem in direct access, I think we can eliminate the possible of problem on DemoAPI-2.

※Code of DemoAPI-1 :

using (var httpClientHandler = new HttpClientHandler())
        {
            httpClientHandler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => { return true; };
            using (var client = new HttpClient(httpClientHandler))
            {
                var req = new HttpRequestMessage(HttpMethod.Get, url);
                req.Headers.Add(HeaderUserAgent, UserAgentValue);
                var response = await client.SendAsync(req).ConfigureAwait(false);
                string buffer = null;
                using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
                {
                    var read = new byte[DataChunkSize];
                    var bytes = stream.Read(read, 0, DataChunkSize);

                    while (bytes > 0)
                    {
                        buffer += System.Text.Encoding.UTF8.GetString(read, 0, bytes);
                        bytes = stream.Read(read, 0, DataChunkSize);
                    }
                }
                return buffer;
            }
        }

※Code of DemoAPI-2 :

[HttpGet(“Demo2”)]

public async Task<IActionResult> Demo2Async(int wait)
  {
      try
      {
          if (wait > 0)
          {
              await Task.Delay(wait);
          }
          var path = Path.Combine(Directory.GetCurrentDirectory(), "test.xml");
          var file = System.IO.File.ReadAllText(path);
          return Content(file);
      }
      catch (System.Exception ex)
      {
          return StatusCode(500, ex.Message);
      }
  }

Additional context

1. Both APIs are async.
2. Both APIs are hosted on different servers (Windows Server 2016).
3. DemoAPI-1(kestrel) is a self-contained API(without reverse proxy)
4. TestClient(jMeter) is set to 100 thread for this testing (want to test upto 1000 threads).
5. No other configuration is done for kestrel server as of now.
6. There are no action filter, middleware or logging that could effect the performance as of now.
7. Communication is done using SSL on 5001 port.
8. Wait parameter for DemoAPI2 is set to 0 as of now.
9. The CPU usage of DEMOAPI-1 is not over 40%.
10. I’ve performed similar test for 100KB and 500KB files and didn’t have any problem.

https://stackoverflow.com/questions/56387959/asp-net-core-2-2-kestrel-servers-performance-issue

Screenshots

I’ve attached the performance monitor’s screenshot.

If this is not the correct place for this question please let me know where should I ask about this.

[dcarr42]

You can try adapting CryptoStream but it seems you have the perf hike from httpclient work.

On Fri, 31 May 2019, 10:40 vishwas-trivedi, [e-mail] wrote:

@DAllanCarr https://github.com/DAllanCarr

Those bytes will be held in memory. I would suggest not buffering the response and read the raw network stream and hand it off to stream reader for processing. For larger requests it could be a consideration. … <#m_-8827034674454892256_> On Fri, 31 May 2019, 08:26 vishwas-trivedi, @.***> wrote: @Tornhoof https://github.com/Tornhoof https://github.com/Tornhoof Thank you! I’m working on a sample to deploy and re-check the same. I’ll share the results soon. Regards, Vishwas — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10681 https://github.com/aspnet/AspNetCore/issues/10681?email_source=notifications&email_token=ADKSKTFLCHXFDC6V46MM2TDPYDHIZA5CNFSM4HRS7XAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWUNXJQ#issuecomment-497605542>, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKSKTDSKYK465XG7ULMP4DPYDHIZANCNFSM4HRS7XAA .

I would be grateful if you could provide a sample. I want to return response in base-64 format from DemoAPI-1 as you can see in my previous comment(#10681 (comment) https://github.com/aspnet/AspNetCore/issues/10681#issuecomment-497600974 ).

Regards, Vishwas

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/AspNetCore/issues/10681?email_source=notifications&email_token=ADKSKTFS3RPZ4UATLZ3HYOTPYDW77A5CNFSM4HRS7XAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWUXYDY#issuecomment-497646607, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKSKTHBA5JG5FHI4WASHBDPYDW77ANCNFSM4HRS7XAA .

[msftbot[bot]]

Thank you for contacting us. Due to a lack of activity on this discussion issue we’re closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn’t been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!