question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

DurableTask.AzureStorage: CreateTimer fails to complete if called after a continue as new

See original GitHub issue

On DurableTask.AzureStorage, the following orchestration hangs; stepping through the debugger reveals that the timer does not fire.

        public override async Task<string> RunTask(OrchestrationContext context, int input)
        {
            if (input == 0)
            {
                context.ContinueAsNew(1);

                return "continue as new";
            }
            else if (input == 1)
            {
                await context.CreateTimer(context.CurrentUtcDateTime, 0);

                return "OK";
            }

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5

github_iconTop GitHub Comments

1reaction
sebastianburckhardtcommented, Jun 18, 2019

I think I understand the cause for the bug, it is indeed related to the fast ContinueAsNew introduced in #251.

What happens is that the loop inside OnProcessWorkItemAsync goes into the second execution of the instance, which has a different execution id, but the execution id in OrchestrationSession is not updated. When the timer fires, the OrchestrationSessionManager executes AddMessageToPendingOrchestration but because the execution id does not match, the message is put back into the queue (with the expectation that the session will disappear and then a new instance would be created by OrchestrationSessionManager.GetNextSessionAsync to handle the message batch). However, this never happens with extended Sessions and fast continueAsNew enabled, so the message is never actually delivered.

I am working on a fix that can update the execution id in OrchestrationSession and deliver the messages to it.

0reactions
cgillumcommented, May 15, 2019

I think this might be caused by the fast-ContinueAsNew change in DurableTask.Core 2.1.0. I believe it may also be causing issues with calling activity functions immediately after ContinueAsNew (some discussion of this here: https://github.com/Azure/azure-functions-durable-extension/issues/529).

Read more comments on GitHub >

github_iconTop Results From Across the Web

Eternal orchestrations stopped randomly · Issue #529
DurableTask.AzureStorage: CreateTimer fails to complete if called after a continue as new Azure/durabletask#285.
Read more >
Handling errors in Durable Functions (Azure Functions)
If the first CreditAccount function call fails, the orchestrator function compensates by crediting the funds back to the source account.
Read more >
Failed to create the task hub: DurableTask.AzureStorage. ...
Net Core 3.1. After that, when I run my functions project it loads all the functions and after loading it fails continuously with...
Read more >
Durable Functions Deep Dive part 3: Running an activity
Crucially, it does not await the Task returned so we can continue processing events. When an activity is called through the orchestration ...
Read more >
Improving scalability and error recovery with Azure Durable ...
The work is tracked by using a dynamic list of tasks. Task.WhenAll is called to wait for all the called functions to finish....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found