Queue trigger timeout exception cause host restart

Similar to https://github.com/Azure/azure-webjobs-sdk/issues/2476, we notice the host is keeping restarting with queue connection issue:

System.Net.Sockets.SocketException A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 52.241.88.73:443

at Microsoft.Azure.Storage.Core.Executor.Executor.<ExecuteAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Extensions.Storage.TimeoutHandler.<ExecuteWithTimeout>d__11.MoveNext() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\TimeoutHandler.cs:line 56 — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ExecuteAsync>d__33.MoveNext() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Queues\Listeners\QueueListener.cs:line 235 — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__14.MoveNext() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\TaskSeriesTimer.cs:line 147 — End of stack trace from previous location where exception was thrown — at Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.<>c__DisplayClass3_0.<OnUnhandledExceptionAsync>b__0() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\WebJobsExceptionHandler.cs:line 54 at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()

Repro steps

Provide the steps required to reproduce the problem None, these errors occur in production env and it is transient. But it had a bad impact.

Expected behavior

The queue connection exception should result to a retry instead of restarting of the host

Actual behavior

web job host keeps restarting

Known workarounds

None

Related information

simular to: https://github.com/Azure/azure-webjobs-sdk/issues/2476

• Package version microsoft.azure.webjobs.extensions.storage\4.0.4

• Links to source https://github.com/Azure/azure-webjobs-sdk/blob/da78eb482e5b2d44fae6b911bc11bddefff7abd7/src/Microsoft.Azure.WebJobs.Extensions.Storage/Queues/Listeners/QueueListener.cs#L235

[DingmaomaoBJTU]

The function app name: BotManagementJob

The function name: BotStateCleanupFunction

Region: Brazil South

Timestamp: 2021-09-22 05:42:42.6912954

[v-anvari]

Closing this issue here as an incident has been submitted to work on the issue