Wait-DurableTask broken in Powershell 7.2 functions

It seems there is inconsistency and breakage with external event handling when using Powershell 7.2 in Azure functions.

HttpTrigger

using namespace System.Net

param($Request, $TriggerMetadata)

$inputs = @{
    Name = "$($Request.Query.Name)"
}

$FunctionName = $Request.Params.FunctionName
$InstanceId = Start-DurableOrchestration -FunctionName $FunctionName -Input $inputs 
Write-Host "Started orchestration with ID = '$InstanceId'"

$Response = New-DurableOrchestrationCheckStatusResponse -Request $Request -InstanceId $InstanceId
Push-OutputBinding -Name Response -Value $Response

Orchestration function

using namespace System.Net
param($Context)

$output = @()
$gate1 = Start-DurableExternalEventListener -EventName "Paris" -NoWait -verbose
$gate2 = Start-DurableExternalEventListener -EventName "London" -NoWait -verbose

$output = Invoke-DurableActivity -FunctionName 'Hello' -Input $Context.Input

$endResults = Wait-DurableTask -Task @($gate1, $gate2)

$finaloutput = Invoke-ActivityFunction -FunctionName 'Bye' -Input $output

$finaloutput

Hello (Activity)

using namespace System.Net
param($name, $TriggerMetadata)

$InstanceId = $TriggerMetadata.InstanceId

Write-Host "Hello $name"

Send-DurableExternalEvent -InstanceId $InstanceId -EventName "London" -verbose
Send-DurableExternalEvent -InstanceId $InstanceId -EventName "Paris" -verbose

$name

Bye (Activity)

using namespace System.Net
param($name)

Write-Host "Bye $name"

$name

Using the above code when I do:

“FUNCTIONS_WORKER_RUNTIME_VERSION”: “~7”

It works and will run Bye and produce the correct output. However, when I change it to:

“FUNCTIONS_WORKER_RUNTIME_VERSION”: “7.2”

It stays in the running state and does not run Bye. Sending the 2 events manually again, it throws an exception below:

 Orchestration completed with a 'Failed' status and 314 bytes of output. Details: Unhandled exception while executing orchestration: DurableTask.Core.Exceptions.NonDeterministicOrchestrationException: Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence ID 1 and name 'Bye' (version ''), but the current replay execution hasn't (yet?) scheduled this task. Was a change made to the orchestrator code after this instance had already started running?
[2022-08-26T06:44:07.415Z]    at DurableTask.Core.TaskOrchestrationContext.HandleTaskScheduledEvent(TaskScheduledEvent scheduledEvent) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 271
[2022-08-26T06:44:07.415Z]    at DurableTask.Core.TaskOrchestrationExecutor.ProcessEvent(HistoryEvent historyEvent) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 189
[2022-08-26T06:44:07.416Z]    at DurableTask.Core.TaskOrchestrationExecutor.<ExecuteCore>g__ProcessEvents|11_0(IEnumerable`1 events) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 114
[2022-08-26T06:44:07.416Z]    at DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore(IEnumerable`1 pastEvents, IEnumerable`1 newEvents) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 122
[2022-08-26T06:44:07.417Z]         at DurableTask.Core.TaskOrchestrationContext.HandleTaskScheduledEvent(TaskScheduledEvent scheduledEvent) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 271
[2022-08-26T06:44:07.417Z]    at DurableTask.Core.TaskOrchestrationExecutor.ProcessEvent(HistoryEvent historyEvent) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 189
[2022-08-26T06:44:07.420Z]    at DurableTask.Core.TaskOrchestrationExecutor.<ExecuteCore>g__ProcessEvents|11_0(IEnumerable`1 events) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 114
[2022-08-26T06:44:07.420Z]    at DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore(IEnumerable`1 pastEvents, IEnumerable`1 newEvents) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 122

If I make the orchestrator slightly complicated:

Orchestration function

using namespace System.Net
param($Context)

$output = @()
$gate1 = Start-DurableExternalEventListener -EventName "Paris" -NoWait -verbose
$gate2 = Start-DurableExternalEventListener -EventName "London" -NoWait -verbose

$output = Invoke-DurableActivity -FunctionName 'Hello' -Input $Context.Input
$output1 = Invoke-DurableActivity -FunctionName 'Hello1' -Input $output

$endResults = Wait-DurableTask -Task @($gate1, $gate2)

$finaloutput = Invoke-ActivityFunction -FunctionName 'Bye' -Input $output1

$finaloutput

Hello1 (Activity)

using namespace System.Net
param($name)

Write-Host "Hello again $name!!"

$name

Then it again does not complete unless I resend the events, but this time I sometimes have to send the events multiple times before it fails again (using ~7 still works though). The more complex the orchestration the more inconsistent it seems to get.

Core Tools Version: 4.0.4736 Commit hash: N/A (64-bit) Function Runtime Version: 4.8.1.18957

When using ~7 it runs PowerShell 7.0.11 When using 7.2 its running PowerShell 7.2.4

Am I doing something wrong? Anyone else have success running external events with PowerShell 7.2 when using complex orchestrations?

[davidmrdavid]

Hey @ruankr:

I just tested this scenario in the new PowerShell Durable Functions SDK announced here and it seems to be working out of the box. Just wanted to notify you in case you wanted to give it a try, but please note it’s still in a public preview stage. Thanks!

[davidmrdavid]

Thanks for the heads up @ruankr. I also seem to be able to repro this with 7.2 but not with ~7.

I’ll keep this thread updated on what happened - I’m putting this as a top priority item.