MessageReceived problems after idle running in Azure

Dotnet Core 2.0.3 Linux host on Azure

I have a bot that I’ve packaged as a Docker container running on Azure Container Instances. I have a very repeatable problem and am currently stuck. After the bot starts, it responds normally to commands I send to it. If I let it sit for about 3 or 4 minutes and send a command, I get no response and see in the log that the gateway is blocked on the MessageHandler. After another minute or so, there is an exception in the log for the task getting canceled and the bot reconnects, after which it is normal again… until I let it sit idle.

It seems like there is a websocket used to POST to Discord’s API that is becoming unusable after being idle for some time. Any idea why this is happening and the best way to resolve it quickly? Waiting for 2 minutes until it comes back to life isn’t going to work. “Debug” log level output is pasted below. I see events coming in the entire time right up to the “A MessageReceived handler…” message. It’s just outbound POSTs that seem to be affected.

03:22:55 Gateway     Received Dispatch (TYPING_START)
03:22:57 Gateway     Received Dispatch (CHANNEL_CREATE)
03:22:57 Gateway     Received Dispatch (MESSAGE_CREATE)
03:23:00 Gateway     A MessageReceived handler is blocking the gateway task.
03:23:27 Gateway     Sent Heartbeat
03:24:08 Gateway     System.Exception: Server missed last heartbeat
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.ConnectionManager.<>c__DisplayClass28_0.<<StartAsync>b__0>d.MoveNext()
03:24:08 Gateway     Disconnecting
03:24:08 Gateway     Disconnecting ApiClient
03:24:36 Gateway     A MessageReceived handler has thrown an unhandled exception.:
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.Net.Rest.DefaultRestClient.<SendInternalAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.Net.Rest.DefaultRestClient.<SendAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.Net.Queue.JsonRestRequest.<SendAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.Net.Queue.RequestBucket.<SendAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.Net.Queue.RequestQueue.<SendAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.API.DiscordRestApiClient.<SendInternalAsync>d__62.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.API.DiscordRestApiClient.<SendJsonAsync>d__59`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.API.DiscordRestApiClient.<CreateMessageAsync>d__77.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.Rest.ChannelHelper.<SendMessageAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.EventExtensions.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Discord.WebSocket.DiscordSocketClient.<TimeoutWrap>d__122.MoveNext()
03:24:36 Gateway     Waiting for heartbeater
03:24:36 Gateway     Waiting for guild downloader
03:24:36 Gateway     Clearing large guild queue
03:24:36 Gateway     Raising virtual GuildUnavailables
03:24:36 Gateway     Disconnected from TESTGUILD1
03:24:36 Gateway     Disconnected from TESTGUILD2
03:24:36 Gateway     Disconnected
03:24:37 Gateway     Connecting
03:24:37 Gateway     Connecting ApiClient
03:24:37 Rest        GET gateway: 36.26 ms
03:24:37 Gateway     Received Hello
03:24:37 Gateway     Heartbeat Started
03:24:37 Gateway     Sent Heartbeat
03:24:37 Gateway     Resuming
03:24:37 Gateway     Sent Resume
03:24:37 Gateway     Received HeartbeatAck
03:24:37 Gateway     Latency = 97 ms
03:24:37 Gateway     Received Dispatch (TYPING_START)
03:24:37 Gateway     Received Dispatch (CHANNEL_CREATE)
03:24:37 Gateway     Received Dispatch (MESSAGE_CREATE)
03:24:38 Rest        POST channels/384397718916694016/messages: 108.53 ms
03:24:38 Gateway     Received Dispatch (RESUMED)
03:24:38 Gateway     Sending Status
03:24:38 Gateway     Sent StatusUpdate
03:24:38 Gateway     Connected
03:24:38 Gateway     Raising Event
03:24:38 Gateway     Connected to TESTGUILD1
03:24:38 Gateway     Connected to TESTGUILD2
03:24:38 Gateway     Resumed previous session
03:24:38 Gateway     Received Dispatch (CHANNEL_CREATE)
03:24:38 Gateway     Received Dispatch (MESSAGE_CREATE)

[cowlike]

I have some good news. I deployed the bot on AWS and it’s running fine there. This seems to be something odd that is going on with Azure causing the bot to lose connection. I’ll look into that separately.

What do you guys think of letting the user specify a timeout value for their own tasks? The bot would still reconnect but it would be much faster. If there was also an event that the timeout happened, the bot could post a message about something that went wrong. Just some thoughts. I’m gong to bug MS about the Azure problem.

[luizcarlosfaria]

For anyone stumbling across this, I recently deployed a bot to azure using Azure Kubernetes Service.

Try out AKS, ACS, Azure Container Instances, or a linux vm and see if that helps resolve the issue.

5 years later, same problem.

On Kubernetes

• Lost connections
• Sometimes I can’t even establish a connection.

On Docker

• Working fine