Performance regression for larger responses in AspNetCore 6.0 with HttpSys and NewtonsoftJson

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When migrating our application from netcoreapp3.1 to net6.0 I noticed a significant performance regression. Our application uses Mvc Core with Newtonsoft Json for serialization/deserialization in requests/responses and is run on HttpSys on Windows. After migration, the net6.0 version is >10 times slower for larger responses (~256KiB).

Performance degradation happens for responses larger than 30KiB. The larger the response, the bigger the degradation.

This reproduces when the app runs on a remote machine. It does not reproduce when running with HttpSys locally (i.e. request to localhost). It also does not reproduce when app runs on a remote machine on Kestrel or IIS in-process or IIS out-of-process.

The issue is somehow related to interaction between FileBufferingWriterStream used by NewtonsoftJsonOutputFormatter and HttpSys. The version used in netcoreapp3.1 calls DrainBufferAsync on response.Body which uses Stream.CopyToAsync(Stream). The version used in net6.0 calls DrainBufferAsync on response.BodyWriter which uses PipeWriter.CopyFromAsync(Stream). The former reads the buffer file in 128KiB chunks, whereas the latter in 4KiB chunks.

If I manually replace NewtonsoftJsonOutputFormatter in net6.0 with my own version that differs only in the call to DrainBufferAsync(response.Body), the issue is resolved.

Expected Behavior

There should be no performance degradation.

Steps To Reproduce

Create a basic web application:

<!-- WebApplication.csproj -->
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp3.1;net6.0</TargetFrameworks>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Condition="'$(TargetFramework)' == 'net6.0'" Version="6.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Condition="'$(TargetFramework)' == 'netcoreapp3.1'" Version="3.1" />
  </ItemGroup>
</Project>

// Program.cs
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace WebApplication
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseHttpSys();
                    webBuilder.UseStartup<Startup>();
                });
    }

    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        public void ConfigureServices(IServiceCollection services)
        {
            var options = services.AddControllers();
            options.AddNewtonsoftJson();
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseRouting();

            app.UseAuthorization();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });
        }
    }

    [ApiController]
    [Route("[controller]")]
    public class TestController : ControllerBase
    {
        [HttpGet]
        public IEnumerable<int> Get(int length = 128)
        {
            length *= 1024;
            return Enumerable.Repeat(0, (length - 1) / 2 - 1)
                .Concat(Enumerable.Repeat(length % 2 == 0 ? 10 : 0, 1));
        }

        [HttpGet("info")]
        public object GetInfo()
        {
            return Environment.Version;
        }
    }
}

Build for netcoreapp3.1 and net6.0:

dotnet publish --configuration Release --self-contained --runtime win-x64 --framework net6.0
dotnet publish --configuration Release --self-contained --runtime win-x64 --framework netcoreapp3.1

Publish to remote machine and query it, e.g. http://[host]/test?length=256 for 256KiB response.

I’m getting the following results for a Windows Azure VM (size D2s v3) in West Europe when querying from Europe:

• netcoreapp 3.1, 256KiB response - ~200ms
• net6.0, 256KiB response - ~2400ms

Exceptions (if any)

No response

.NET Version

6.0.2

Anything else?

No response

[davidfowl]

Tweaking the buffer size might be a better solution in your scenario. We avoid the LOH by using pooled and paged memory under the covers. So it shouldn’t be a concern, here.

[msftbot[bot]]

Thanks for contacting us.

We’re moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it’s very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.