Timeouts finish unexpectedly when events are fired inside an orchestration

As part of a demo, I have a sub-orchestration that simulates a 2 step communication with a 3rd party. (sends a signal and then waits for a callback until continuing with the execution. The issue is that every time an event is raised, the timeOut suddenly completes during the initial replay (so before the onEvent was processed) and the suborchestration fails due to this. I’m using the SQLProvider for DTF.

Sample code : Sub orchestration :

public class CompatibilityOrchestrator : TaskOrchestration<CompatibilityResponse, CompatibilityGenerationRequest, CompatibilityResponse, string>
    {
        TaskCompletionSource<CompatibilityResponse> receivedCompatResponseEvent = new TaskCompletionSource<CompatibilityResponse>();
        private readonly ILogger<CompatibilityOrchestrator> _logger;

        public CompatibilityOrchestrator(ILogger<CompatibilityOrchestrator> logger)
        {
            _logger = logger;
        }
       public override async Task<CompatibilityResponse> RunTask(OrchestrationContext context, CompatibilityGenerationRequest input)
        {
            var sv = input;
            sv.OrchestrationInstanceID = context.OrchestrationInstance.InstanceId;
            _logger.LogInformation("Sending generate compatibility report signal to 3rd Party :" + sv.OrchestrationInstanceID.ToString());

            var timeoutTask = context.CreateTimer(context.CurrentUtcDateTime.AddMinutes(5), "TimedOut");

            var winner = await Task.WhenAny(receivedCompatResponseEvent.Task, timeoutTask);

            if (winner == receivedCompatResponseEvent.Task && receivedCompatResponseEvent.Task.Result != null)
            {
                return receivedCompatResponseEvent.Task.Result;
            }
            else
            {
                throw new TimeoutException();
            }
        }

        public override void OnEvent(OrchestrationContext context, string name, CompatibilityResponse compatResponse)
        { 
           if (name.Equals("ReceiveCompatResponseEvent")) { 
                receivedCompatResponseEvent.SetResult(compatResponse);
            }
        }
    }

The event is raised from the following endpoint :

        [HttpPost]
        [Route("api/{partitionId}/compatResponse")]
        public async Task CompatibilityReportResponse([FromRoute] string partitionId, [FromBody] CompatibilityResponse data)
        {
            await _workflowClient.Client.RaiseEventAsync(new OrchestrationInstance() { InstanceId = data.OrchestrationInstanceID}, "ReceiveCompatResponseEvent",data);
        }

The issue is that no matter how long I set the expiration of the timer (1 day, 1 month), whenever I fire an event that triggers this subOrchestration, the timer completes and the subOrchestration fails with TimeoutException(); I was able to work around this by creating a custom activity that simulates “waiting”, and then waiting for that task instead of the original timer :

public class FakeTimerActivity : TaskActivity<string, Task<string>>
    {
        private readonly ILogger<FakeTimerActivity> _logger;

        public FakeTimerActivity(ILogger<FakeTimerActivity> logger)
        {
            _logger = logger;
        }

        protected override Task<string> Execute(TaskContext context, string input)
        {
            return Task.Delay(50000).ContinueWith(t => "Hello");
        }
    }

And in the orchestration I would use this : var timeoutTask = context.ScheduleTask<string>(typeof(FakeTimerActivity));

[moldovangeorge]

After taking a look at the history of the orchestration in the DB I can see that a TimerFire event is processed before the event is raised : The timestamp of the TimerFire is the same as the one of TimerCreated but that is not true because the TimerFire happened when the event was raised ( I investigated the db before, during and after the orchestration finished). The flow seems to be this one :

• Orchestration is created and a Timer Event is registered in the NewEvents table -> The VisibleTime is set correctly ( a dateTime in the future)
• Orchestration halts and waits for either the Timeout or the EventTask to finish
• Event is raised and this makes the Timer Event to be consumed from the NewEvents table, even though the VisibleTime is still in the future -> a TimerFire event is logged in the History table and the Timeout task is completed
• The Orchestration replays, finds out that the Timeout task is completed and exits.
• The event that was raised is processed but the orchestration is already finished.

[cgillum]

Thanks for these details. It sounds like I may have misunderstood the original issue. What you’re describing sounds a lot like https://github.com/microsoft/durabletask-mssql/issues/50.