Duplex streaming problems like: The client reset the request stream.

Hello, in our production application we have gRPC

• server using ASP.NET Core 5 implementation
• .NET Framework 4.8 clients using C# Grpc.Core implementation

Duplex streaming calls are used in various cases like

• notifications about data changes
• image streaming

Clients call the server, stream requests, read streamed responses and close request stream.

Sometimes we see a call ends with errors on server in requestStream.MoveNext, but has no problem on clients. This is not problematic when we do all processing during streaming and error on server means actually a client has ended the call. But this is problematic in cases when we do some processing on server after a client call end, but due to the error the processing is skipped and client just thinks everything is ok, which is not obviously.

To reproduce the problem, I have prepared testing service and client attached to this post:

• service uses ASP.NET Core 5 implementation
• client may choose .NET or Grpc.Core implementation, but both show the problem

Steps to reproduce the problem:

• start service
• start client and choose .NET or Grpc.Core implementation
• client calls the service duplex streaming call with short delay (100 ms which can be changed with ± keys) between responses
• service processes the call with short delay (100 ms which can be changed with ± keys) between responses
• to end the call on client and start another one, press any key (eg. space)
• depending on the call duration, the call ends well or with errors in the service - just press a key on client few times after various times
• to continue the service after any error, press Enter

The errors I have seen are:

• info: Grpc.AspNetCore.Server.ServerCallHandler[14] Error reading message. System.IO.IOException: The client reset the request stream. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.AdvanceTo(SequencePosition consumed, SequencePosition examined) at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func2 deserializer, CancellationToken cancellationToken) System.IO.IOException: The client reset the request stream. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.AdvanceTo(SequencePosition consumed, SequencePosition examined) at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func2 deserializer, CancellationToken cancellationToken) at Grpc.AspNetCore.Server.Internal.HttpContextStreamReader1.<MoveNext>g__MoveNextAsync|11_0(ValueTask1 readStreamTask) at Service.GreeterService.SayHello(IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream, ServerCallContext context) in GrpcDuplexStreaming\Service\Services\GreeterService.cs:line 21
• System.InvalidOperationException: Can’t read messages after the request is complete. at Service.GreeterService.SayHello(IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream, ServerCallContext context) in GrpcDuplexStreaming\Service\Services\GreeterService.cs:line 21

I consider this as severe problem and would like to see any fix or recommendation how to deal with it.

Thank You

Marek Ištvánek

[JamesNK]

~I think I found another issue on the server. It will require a fix in .NET which will take a while.~ Edit: No, I think the server is fine. Basically, if you don’t gracefully complete a request (described below) then you’re setting yourself up for errors.

What I recommend you do is modify the client to gracefully complete both sides of the stream before disposing the call. That is best practice when doing streams.

To do this the client calls RequestStream.CompleteAsync(), it then waits for ResponseStream.MoveNext() to return false.

[istvanek]

@JamesNK sure. I will create another issue for the last case when I will get more time for replication sample. Thank You.