Low throughput with large messages and high latency (25ms RTT)

Greetings!

I am having performance issues receiving/sending large messages (~100k-100MB) over a connection with 25 ms RTT. Transfer rate is ~2 MB/s but when serving static files of the same size on the same kestrel instance >25 MB/s (tested in Chrome with stopwatch). When testing on localhost, 200 MB/s.

I see the issue both with large unary operations or streaming (breaking up into chunks between 10K and 10MB). Poking about in WireShark, it looks like the large static downloads dramatically increase the TCP window to compensate for the latency during the transfer while gRPC transfers don’t. I assume this is the issue. I have not been able to find any settings which change this.

Any ideas? Thanks!

The Service

public override Task<DataAccessResponse> TestDownload(DataAccessRequest request, ServerCallContext context)
{
    _bigchunk ??= new byte[100_000_000];

    var response = new DataAccessResponse
    {
        DataChunk = ByteString.CopyFrom(_bigchunk)
    };
    
    return Task.FromResult(response);
}

Server Setup

  services.AddGrpc(grpc =>
  {
      grpc.MaxReceiveMessageSize = 100_000_000;
  });

 webBuilder.UseKestrel(options =>
  {
      //I have used the defaults and played around with these settings - no difference
      options.Limits.Http2.InitialConnectionWindowSize = int.MaxValue;
      options.Limits.Http2.InitialStreamWindowSize = int.MaxValue;
      options.Limits.Http2.MaxFrameSize = 16_777_215;
      options.AllowSynchronousIO = true;
  });
}

Dependencies

    <PackageReference Include="Google.Protobuf" Version="3.13.0" />
    <PackageReference Include="Grpc.AspNetCore.Server" Version="2.35.0" />
    <PackageReference Include="Grpc.Net.Common" Version="2.35.0" />
    <PackageReference Include="Grpc.Net.ClientFactory" Version="2.35.0" />
    <PackageReference Include="Grpc.Core.Api" Version="2.35.0" />

.NET 5.0

[davidfowl]

BTW, googled you and realized we are both Florida Tech alums (although back in the stone ages we called it FIT).

Panthers unite!

[SartorialOffense]

Thanks for the help, I have workarounds until this is fixed in hopefully 6.0.