AzureStorage: Large orchestrationInstances are corrupted, causing discarded messages

Large orchestrationInstances (>~16kb) are being sent to table storage as a DurableTask.Core.History.GenericEvent rather than an ExecutionStartedEvent. When the framework pulls the history event, it is discarded with this error: “Discarding 1 dequeued history event(s): Instance is corrupted.” The corruption seems to be because the framework cannot find an ExecutionStartedEvent. The issue is not specific to the activity/orchestration being sent, just on the size of the data. We checked to see if it was a specific range of sizes, but this occurs for a message of about 150kb as well.

This issue causes any orchestrations with enough data to fail somewhat silently – the first activity will execute, then the task completed message will be discarded without any exception or indication that orchestration size is the issue.

Is this intended behavior? We would expect that we would receive an exception if the orchestration instance is too large, but not sure if we are perhaps doing something wrong here.

Type and the start of data for a message this occurs for:

"$type": "DurableTask.Core.History.GenericEvent, DurableTask.Core",
"Data": "{\r\n  \"$type\": \"DurableTask.Core.History.ExecutionStartedEvent, DurableTask.Core\",\r\n  \"OrchestrationInstance\":

The exact same orchestration type, but with slightly less data in the actual OrchestrationInstance

"$type": "DurableTask.Core.History.ExecutionStartedEvent, DurableTask.Core",
"OrchestrationInstance":

Happy to provide more information/examples to assist with debugging

[cgillum]

Hi @vany0114. You mentioned you’re using a JS SDK. Would you mind opening an issue here so that we can get the right folks to take a look? https://github.com/Azure/azure-functions-durable-js/issues. FYI @davidmrdavid

BTW, Connor is no longer working on the project, but I think David should be able to help.

[vany0114]

@cgillum @ConnorMcMahon I’m experiencing this issue, I’m using the JS SDK to orchestrate a couple of timers, this error is causing the timer never to get fired off, any idea?

Thanks in advance.