Singleton Listeners causing 409 (Conflict) error

Please provide a succinct description of the issue.

When you run timer jobs with many worker roles (50+) one of those worker roles gets the 409 error from storage service and gets recycled. When this happens the exception message says “StatusMessage:The lease ID specified did not match the lease ID for the blob. ErrorCode:LeaseIdMismatchWithLeaseOperation”.

Provide the steps required to reproduce the problem

10 timer jobs with 50-150 worker role instances. Every 30 minutes also one of roles will start recycling due to 409 (conflict) error.

Expected behavior

No conflict error from storage service.

Provide a description of the expected behavior.

Actual behavior

Getting 409 error.

Known workarounds

None.

Related information

Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (409) Conflict. —> System.Net.WebException: The remote server returned an error: (409) Conflict. at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase1 cmd, Exception ex) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Shared\Protocol\HttpResponseParsers.Common.cs:line 50 at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.<>c__DisplayClass33.<RenewLeaseImpl>b__32(RESTCommand1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 3319 at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 306 — End of inner exception stack trace — at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 60 at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 115 — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.SingletonManager.RenewLeaseCommand.<ExecuteAsync>d__36.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() Request Information RequestID:d8d712e5-0001-012c-651a-0dec95000000 RequestDate:Mon, 12 Sep 2016 17:26:41 GMT StatusMessage:The lease ID specified did not match the lease ID for the blob. ErrorCode:LeaseIdMismatchWithLeaseOperation

[alohaninja]

seeing this 409 LeaseIdMismatchWithLeaseOperation regularly with our blob trigger. We have very low volume and only (2) instances.

2017-11-28T03:28:58.536 Host lock lease acquired by instance ID '66525d2801dbf933e2e11518b127855f'.
info: Host.General[0]
      Host lock lease acquired by instance ID '66525d2801dbf933e2e11518b127855f'.
2017-11-28T03:29:06.300 A ScriptHost error has occurred
2017-11-28T03:29:06.300 System.InvalidOperationException : Singleton lock renewal failed for blob 'locks/my-function/host' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 0001-01-01T00:00:00Z (-2147483648 milliseconds ago) with a duration of 0 milliseconds. The lease period was 15000 milliseconds.
2017-11-28T03:29:06.309 	 'locks/my-function/host' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 0001-01-01T00:00:00Z (-2147483648 milliseconds ago) with a duration of 0 milliseconds. The lease period was 15000 milliseconds.
2017-11-28T03:29:06.312 Failed to renew host lock lease: Another host has acquired the lease. The last successful renewal completed at 2017-11-28T03:28:54.229Z (12080 milliseconds ago) with a duration of 780 milliseconds.

Seems to freeze up our host and we have to restart the function to get it working again.

[abatishchev]

I’m facing the same error when debug locally a webjob having several scheduled singleton functions.