DurableTask.AzureStorage: CreateTimer fails to complete if called after a continue as new
See original GitHub issueOn DurableTask.AzureStorage, the following orchestration hangs; stepping through the debugger reveals that the timer does not fire.
public override async Task<string> RunTask(OrchestrationContext context, int input)
{
if (input == 0)
{
context.ContinueAsNew(1);
return "continue as new";
}
else if (input == 1)
{
await context.CreateTimer(context.CurrentUtcDateTime, 0);
return "OK";
}
Issue Analytics
- State:
- Created 4 years ago
- Comments:5
Top Results From Across the Web
Eternal orchestrations stopped randomly · Issue #529
DurableTask.AzureStorage: CreateTimer fails to complete if called after a continue as new Azure/durabletask#285.
Read more >Handling errors in Durable Functions (Azure Functions)
If the first CreditAccount function call fails, the orchestrator function compensates by crediting the funds back to the source account.
Read more >Failed to create the task hub: DurableTask.AzureStorage. ...
Net Core 3.1. After that, when I run my functions project it loads all the functions and after loading it fails continuously with...
Read more >Durable Functions Deep Dive part 3: Running an activity
Crucially, it does not await the Task returned so we can continue processing events. When an activity is called through the orchestration ...
Read more >Improving scalability and error recovery with Azure Durable ...
The work is tracked by using a dynamic list of tasks. Task.WhenAll is called to wait for all the called functions to finish....
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I think I understand the cause for the bug, it is indeed related to the fast
ContinueAsNew
introduced in #251.What happens is that the loop inside
OnProcessWorkItemAsync
goes into the second execution of the instance, which has a different execution id, but the execution id inOrchestrationSession
is not updated. When the timer fires, theOrchestrationSessionManager
executesAddMessageToPendingOrchestration
but because the execution id does not match, the message is put back into the queue (with the expectation that the session will disappear and then a new instance would be created byOrchestrationSessionManager.GetNextSessionAsync
to handle the message batch). However, this never happens with extended Sessions and fast continueAsNew enabled, so the message is never actually delivered.I am working on a fix that can update the execution id in
OrchestrationSession
and deliver the messages to it.I think this might be caused by the fast-ContinueAsNew change in DurableTask.Core 2.1.0. I believe it may also be causing issues with calling activity functions immediately after ContinueAsNew (some discussion of this here: https://github.com/Azure/azure-functions-durable-extension/issues/529).