Non-Deterministic workflow detected: TaskScheduledEvent: 0 TaskScheduled CopyIModelToProject

I have recently run into an issue with the “Non-Deterministic workflow detected” message while using Durable functions with Node.js + Typescript. (Note: I am on a consumption plan). The workflow that leads to this exception involves a few components:

• A Http Endpoint function ‘SampleIModelProjects’ with GET, POST, PATCH, & DELETE
• A Http Orchestration Starter (Starts Orchestration Client -> Orchestration Trigger)
• An Orchestrator (Activated by Orchestration Trigger -> Activity Trigger)
• An Activity Function ‘CopyIModelToProject’ (Long running, Memory/CPU intensive, with async I/O)(Activated by Activity Trigger) (Usually takes 1-5 minutes)
• An Http Orchestration Status Endpoint

When POST is called on the ‘SampleIModelProjects’ it calls a few APIs, posts to start the Http Orchestration Starter, and returns some data regarding the created project and the status URI for the Orchestration that was started. This works fine in most cases when the load scales slowly, but when the load scales to quickly (lets say 5 workflows in 10 seconds) is when I get the exception “Non-Deterministic workflow detected” and the CopyIModelToProject fails. See below for more details, any suggestions are welcome.

This is what the Status endpoint then returns with 500 when this happens: Error: StatusCodeError: 500 - {“instanceId”:“2ed8b845-b2fc-42c4-9a54-f27c46e4eae3”,“runtimeStatus”:“Failed”,“input”:{“targetProjectName”:“IMODEL_DEV_SERVICES_LOAD_TEST”,“srcProjectName”:“Test Project”,“srcIModelName”:“Test IModel”,“userEmails”:[“[e-mail]”],“targetProjectId”:“f7c35bdd-eb24-4450-8273-e2594162c1e5”},“customStatus”:null,“output”:“Non-Deterministic workflow detected: TaskScheduledEvent: 0 TaskScheduled CopyIModelToProject”,“createdTime”:“2018-11-14T17:45:51Z”,“lastUpdatedTime”:“2018-11-14T17:48:11Z”}

This is what is logged in App Insights:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.CopyIModelToProject ---> Microsoft.Azure.WebJobs.Script.Rpc.RpcException: Result: Failure
Exception: Error Copying IModel
Stack: 
   at Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Rpc\WorkerLanguageInvoker.cs:line 74
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs:line 84
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionGenerator.cs:line 225
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 63
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 556
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 503
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, ParameterHelper parameterHelper, IFunctionOutputDefinition outputDefinition, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 439
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 249
   --- End of inner exception stack trace ---
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 293
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 89

Here is my host.json:

{
    "version": "2.0",
    "functionTimeout": "00:10:00",
    "extensions": {
        "http": {
            "routePrefix": "api",
            "maxOutstandingRequests": 100,
            "maxConcurrentRequests": 50,
            "dynamicThrottlesEnabled": false
        },
        "queues": {
            "maxPollingInterval": 60000,
            "visibilityTimeout" : "00:00:30"
        },
        "durableTask": {
            "HubName": "DurableTaskHub",
            "extendedSessionsEnabled": true,
            "ControlQueueBatchSize": 32,
            "PartitionCount": 4,
            "ControlQueueVisibilityTimeout": "00:01:00",
            "WorkItemQueueVisibilityTimeout": "00:01:00",
            "MaxConcurrentActivityFunctions": 3,
            "MaxConcurrentOrchestratorFunctions": 3
        }
    },
    "logger": {
        "categoryFilter": {
            "defaultLevel": "Information",
            "categoryLevels": {
                "Host": "Error",
                "Function": "Error",
                "Host.Aggregator": "Information"
            }
        },
        "applicationInsights": {
            "sampling": {
              "isEnabled": true,
              "maxTelemetryItemsPerSecond" : 5
            }
        }
    }
  }

[cgillum]

The fix for this issue has been merged and will be available in the next release.

[leo-buneev]

I’m using JS and having this issue with any unhandled exception whatsoever, even in simplest cases.

Microsoft.Azure.WebJobs.Extensions.DurableTask version 1.8.3 (tried with 1.7.0 and 2.0.0-beta1 too with no luck)

Orchestrator:

durableFunctions.orchestrator(function*(context) {
  const args = context.df.getInput()
  yield context.df.callActivity('activity', args)
})

Activity:

function(context) {
  throw new Error('Test error')
  context.done()
}

Results in Non-Deterministic workflow detected: TaskScheduledEvent: 0 TaskScheduled activity

the only way to get rid of it - always put try/catch for entire orchestration code. InstanceId 2ebb9d195cfc4b08946efe3ac5f5ed6a