SignalrManager has trouble reconnecting, failing with timeout

Not super sure what’s going on here, so I’ll start with what I’m seeing. After a disconnect, the signalr socket fails to reconnect with a timeout. The socket connection looks healthy in fiddler (we are receiving pong events).

In the callback attached to the customTransport.WebSocket.Connected event, multiple threads enter which seems weird because I’d only expect 1 StartAsync. (Maybe every socket that failed to reconnect also tries to reconnect? not sure…)

        public async Task StartAsync()
        {
            // stop any previous hub connection
            hubConnection?.Stop();
            hubConnection?.Dispose();

            // make a new hub connection
            hubConnection = new HubConnection(ConnectionUrl, false);
            hubConnection.Closed += SocketClosed;

#if DEBUG

            hubConnection.TraceLevel = TraceLevels.All;
            hubConnection.TraceWriter = Console.Out;

#endif

            hubProxy = hubConnection.CreateHubProxy(HubName);
            if (hubProxy == null)
            {
                throw new APIException("CreateHubProxy - proxy is null, this should never happen");
            }

            // assign callbacks for events
            foreach (string key in FunctionNamesToFullNames.Keys)
            {
                hubProxy.On(key, async (string data) => await HandleResponse(key, data));
            }

            // create a custom transport, the default transport is really buggy
            DefaultHttpClient client = new DefaultHttpClient();
            customTransport = new WebsocketCustomTransport(client);
            var autoTransport = new AutoTransport(client, new IClientTransport[] { customTransport });
            hubConnection.TransportConnectTimeout = hubConnection.DeadlockErrorTimeout = TimeSpan.FromSeconds(10.0);

            // setup connect event
            customTransport.WebSocket.Connected += async (ws) =>
            {
                IWebSocket[] socketsCopy;
                lock (sockets)
                {
                    socketsCopy = sockets.ToArray();
                }
                foreach (IWebSocket socket in socketsCopy)
                {
                    await (socket as SignalrSocketConnection).InvokeConnected();
                }
            };

18:46:14.0036320 - 8ef82e72-d27d-446b-9ac3-ee3c190cd0d4 - Abort never fired
18:46:14.0046303 - 8ef82e72-d27d-446b-9ac3-ee3c190cd0d4 - Disconnected
18:46:14.0086188 - 8ef82e72-d27d-446b-9ac3-ee3c190cd0d4 - Transport.Dispose(8ef82e72-d27d-446b-9ac3-ee3c190cd0d4)
18:46:14.0495095 - 8ef82e72-d27d-446b-9ac3-ee3c190cd0d4 - Closed
18:46:14.1332856 - null - ChangeState(Disconnected, Connecting)
ClientWebSocket.ReadTask: The operation was canceled.
18:46:29.4154326 - 9b456e02-f4d1-480b-8647-bdd9207184f3 - Auto: Failed to connect to using transport webSockets. System.TimeoutException: Transport timed out trying to connect
ClientWebSocket.ReadTask: The operation was canceled.
18:46:29.4872407 - 9b456e02-f4d1-480b-8647-bdd9207184f3 - Disconnected
18:46:29.4882381 - 9b456e02-f4d1-480b-8647-bdd9207184f3 - Transport.Dispose(9b456e02-f4d1-480b-8647-bdd9207184f3)
18:46:29.4922271 - 9b456e02-f4d1-480b-8647-bdd9207184f3 - Closed
19:07:27.2229241 - null - Error: System.TimeoutException: Transport timed out trying to connect
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<StartAsync>d__28.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 519
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<ReconnectLoop>d__26.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 411
19:07:27.4413405 - null - Error: System.TimeoutException: Transport timed out trying to connect
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<StartAsync>d__28.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 519
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<ReconnectLoop>d__26.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 411
19:07:27.4543056 - null - ChangeState(Disconnected, Connecting)
ClientWebSocket.ReadTask: The remote party closed the WebSocket connection without completing the close handshake.
19:07:47.8383033 - 4f3ff6f7-79eb-4ca4-8a1a-9482f9a09e3e - Auto: Failed to connect to using transport webSockets. System.TimeoutException: Transport timed out trying to connect
19:07:47.8442869 - 4f3ff6f7-79eb-4ca4-8a1a-9482f9a09e3e - Disconnected
19:07:47.8452849 - 4f3ff6f7-79eb-4ca4-8a1a-9482f9a09e3e - Transport.Dispose(4f3ff6f7-79eb-4ca4-8a1a-9482f9a09e3e)
19:07:47.8452849 - 4f3ff6f7-79eb-4ca4-8a1a-9482f9a09e3e - Closed
ClientWebSocket.ReadTask: The operation was canceled.
19:07:56.2132244 - null - Error: System.TimeoutException: Transport timed out trying to connect
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<StartAsync>d__28.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 519
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<ReconnectLoop>d__26.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 411
19:07:56.2431437 - null - Error: System.TimeoutException: Transport timed out trying to connect
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<StartAsync>d__28.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 519
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ExchangeSharp.SignalrManager.<ReconnectLoop>d__26.MoveNext() in D:\Git\CryptoArbs\ExchangeSharp\ExchangeSharp\API\Common\SignalrManager.cs:line 411

[bichuga]

oops this one is my fault. I removed the connected callback that cleared all the order books because this reconnect issue kept flushing everything. Either way the code should be

            internal async Task InvokeConnected()
            {
                if (Connected != null)
                {
                    await Connected.Invoke(this);
                }
            }

[ridingmotorcycleslove]

You said the code should be ?