gRPC-web stops communicating after the first communication of a large string submission

What version of gRPC and what language are you using?

C#

Server’s configuration:

    <PackageReference Include="Grpc.AspNetCore" Version="2.30.0" />
    <PackageReference Include="Grpc.AspNetCore.Web" Version="2.30.0" />

Client’s configuration:

  <ItemGroup>
    <PackageReference Include="Google.Protobuf" Version="3.12.3" />
    <PackageReference Include="Grpc.Net.Client" Version="2.30.0" />
    <PackageReference Include="Grpc.Net.Client.Web" Version="2.30.0" />
    <PackageReference Include="Grpc.Tools" Version="2.30.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

What operating system (Linux, Windows,…) and version?

Azure App Services (Windows) Development environment: Windows 10, Visual Studio 2019

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.net core info: C:\Temp>dotnet --info .NET Core SDK (reflecting any global.json): Version: 3.1.302 Commit: 41faccf259

Runtime Environment: OS Name: Windows OS Version: 10.0.17763 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\3.1.302\

Host (useful for support): Version: 3.1.6 Commit: 3acd9b0cd1

.NET Core SDKs installed: 3.1.102 [C:\Program Files\dotnet\sdk] 3.1.202 [C:\Program Files\dotnet\sdk] 3.1.302 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

What did you do?

We followed the same pattern as the Browser App suggests to setup a gRPC-web service. The client code which is a C# code prepares and submits a 2,937,558 bytes string to the gRPC-web application. Submitting the string to the gRPC-web app on development machine is consistently successful. Getting the same string submitted to the same app hosted on Azure results into success ONLY in the first call and all subsequent calls consistently fail. To recover from this situation, we have to restart the App Service on Azure. We also scaled up the service to P3V2 plan but that did not lead to a success.

What did you expect to see?

The gRPC-web app hosted on Azure should not fail in the subsequent calls.

What did you see instead?

Failure and exception in the client application per the following stack trace:

dbug: Grpc.Net.Client.Internal.GrpcCall[1] Starting gRPC call. Method type: ‘Unary’, URI: ‘https://MASKED1234.azurewebsites.net/MASKED1234.PayloadSubmission/Submit’. dbug: Grpc.Net.Client.Internal.GrpcCall[18] Sending message. fail: Grpc.Net.Client.Internal.GrpcCall[20] Error sending message. System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request… —> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. — End of inner exception stack trace — at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory1 source) at System.Net.Http.HttpConnection.ChunkedEncodingWriteStream.<WriteAsync>g__WriteChunkAsync|2_0(HttpConnection connection, ReadOnlyMemory1 buffer) at Grpc.Net.Client.Web.Internal.Base64RequestStream.WriteAsync(ReadOnlyMemory1 data, CancellationToken cancellationToken) at Grpc.Net.Client.StreamExtensions.WriteMessageAsync[TMessage](Stream stream, ILogger logger, TMessage message, Action2 serializer, String grpcEncoding, Nullable1 maximumMessageSize, Dictionary2 compressionProviders, CallOptions callOptions) fail: Grpc.Net.Client.Internal.GrpcCall[6] Error starting gRPC call. System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory1 source) at System.Net.Http.HttpConnection.ChunkedEncodingWriteStream.<WriteAsync>g__WriteChunkAsync|2_0(HttpConnection connection, ReadOnlyMemory1 buffer) at Grpc.Net.Client.Web.Internal.Base64RequestStream.WriteAsync(ReadOnlyMemory1 data, CancellationToken cancellationToken) at Grpc.Net.Client.StreamExtensions.WriteMessageAsync[TMessage](Stream stream, ILogger logger, TMessage message, Action2 serializer, String grpcEncoding, Nullable1 maximumMessageSize, Dictionary2 compressionProviders, CallOptions callOptions) at Grpc.Net.Client.Internal.PushUnaryContent`2.WriteMessageCore(ValueTask writeMessageTask) at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask) — End of inner exception stack trace — at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask) at Grpc.Net.Client.Web.Internal.GrpcWebRequestContent.SerializeTextToStreamAsync(Stream stream) at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stre

Anything else we should know about your project / environment?

The gRPC-web application was deployed as a self-contained application to Azure. The server does not log any failures or exceptions.

[Harkole]

@JamesNK it only seems to affect IIS, I have seen it on Windows Server 2016 and on Application service plan running Windows as the host

Anything using Kestrel is fine as far as I have managed to test

Is there something that needs configured under IIS to enable support of the gRPC-web?

[Arash-Sabet]

@Harkole Yes, I will give that a try and let you know the results, hopefully by the end of this week or mid next week.