Connection to Azure service drops and cannot automatically recover

Yesterday our web applications had connections to the Azure SignalR service dropped. We got several errors like this:

Connection "0a2f27dc-80e1-451e-aa4d-999d4bb950c6" to the service was dropped.

System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
   at async Task<WebSocketReceiveResult> System.Net.WebSockets.WebSocketBase+WebSocketOperation.Process(Nullable<ArraySegment<byte>> buffer, CancellationToken cancellationToken)
   at async Task<WebSocketReceiveResult> System.Net.WebSockets.WebSocketBase.ReceiveAsyncCore(ArraySegment<byte> buffer, CancellationToken cancellationToken)
   at async Task Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport.StartReceiving(WebSocket socket)
   at void System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at void System.IO.Pipelines.Pipe.GetReadResult(ref ReadResult result)
   at ReadResult System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at async Task Microsoft.Azure.SignalR.ServiceConnectionBase.ProcessIncomingAsync()

It looked like one application instance dropped all connections and the other only dropped one.

Then we got several thousand of these errors:

Error while sending message to the service.

Microsoft.Azure.SignalR.Common.ServiceConnectionNotActiveException: The connection is not active, data cannot be sent to the service.
   at async Task Microsoft.Azure.SignalR.ServiceConnectionBase.WriteAsync(ServiceMessage serviceMessage)
   at async Task Microsoft.Azure.SignalR.ServiceConnection.ProcessOutgoingMessagesAsync(ServiceConnectionContext connection)

Then several thousand of these:

Failed writing message. Aborting connection.

System.InvalidOperationException: Writing is not allowed after writer was completed.
   at void System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoWritingAllowed()
   at Memory<byte> System.IO.Pipelines.Pipe.GetMemory(int sizeHint)
   at Span<byte> System.IO.Pipelines.Pipe+DefaultPipeWriter.GetSpan(int sizeHint)
   at ValueTask<FlushResult> System.IO.Pipelines.PipeWriter.WriteAsync(ReadOnlyMemory<byte> source, CancellationToken cancellationToken)
   at async Task Microsoft.AspNetCore.SignalR.HubConnectionContext.TryWritePingSlowAsync()

This went on for over an hour until the web applications were manually restarted. Then the apps were able to re-connect and things were fine.

Versions: JS client: 1.1.0 Microsoft.AspNetCore.SignalR: 1.1.0 Microsoft.Azure.SignalR: 1.0.6

[NArnott]

This seems to be happening twice a week for us, forcing us to restart our service that connects to Azure SignalR.

Microsoft.Azure.SignalR.Common.ServiceConnectionNotActiveException: The connection is not active, data cannot be sent to the service.
   at Microsoft.Azure.SignalR.ServiceConnectionBase.WriteAsync(ServiceMessage serviceMessage)
   at Microsoft.Azure.SignalR.ServiceConnection.ProcessOutgoingMessagesAsync(ServiceConnectionContext connection)

We’re currently using Microsoft.Azure.SignalR version 1.0.9. This is on a Standard Tier, 1 unit Azure SignalR instance.

This is our startup code:

public void ConfigureServices(IServiceCollection services)
{
...

            var signalRBuilder = services.AddSignalR();
            if (!Env.IsDevelopment())
            {
                signalRBuilder.AddAzureSignalR(x =>
                {
                    x.ConnectionString = Configuration.GetConnectionString("AzureSignalR");

                    //Seperate out each environment into a seperate SignalR service
                    x.ApplicationName = Env.EnvironmentName;

                    if (!Env.IsProduction())
                    {
                        //We don't need the default 5 connections for test environments.
                        x.ConnectionCount = 1;
                    }
                });
            }
...
}

public void Configure(IApplicationBuilder app)
{
...
            if (Env.IsDevelopment())
            {
                app.UseSignalR(x =>
                {
                    x.MapHub<PushNotificationsHub>("/hubs/pushNotifications");
                });
            }
            else
            {
                app.UseAzureSignalR(x =>
                {
                    x.MapHub<PushNotificationsHub>("/hubs/pushNotifications");
                });
            }
...
}

[vicancy]

@yosigolan Connection limits are client+server connections in total, that is 20.