Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

AzureStorage: Large orchestrationInstances are corrupted, causing discarded messages

See original GitHub issue

Large orchestrationInstances (>~16kb) are being sent to table storage as a DurableTask.Core.History.GenericEvent rather than an ExecutionStartedEvent. When the framework pulls the history event, it is discarded with this error: “Discarding 1 dequeued history event(s): Instance is corrupted.” The corruption seems to be because the framework cannot find an ExecutionStartedEvent. The issue is not specific to the activity/orchestration being sent, just on the size of the data. We checked to see if it was a specific range of sizes, but this occurs for a message of about 150kb as well.

This issue causes any orchestrations with enough data to fail somewhat silently – the first activity will execute, then the task completed message will be discarded without any exception or indication that orchestration size is the issue.

Is this intended behavior? We would expect that we would receive an exception if the orchestration instance is too large, but not sure if we are perhaps doing something wrong here.

Type and the start of data for a message this occurs for:

"$type": "DurableTask.Core.History.GenericEvent, DurableTask.Core",
"Data": "{\r\n  \"$type\": \"DurableTask.Core.History.ExecutionStartedEvent, DurableTask.Core\",\r\n  \"OrchestrationInstance\":

The exact same orchestration type, but with slightly less data in the actual OrchestrationInstance

"$type": "DurableTask.Core.History.ExecutionStartedEvent, DurableTask.Core",

Happy to provide more information/examples to assist with debugging

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

cgillumcommented, Feb 8, 2022

Hi @vany0114. You mentioned you’re using a JS SDK. Would you mind opening an issue here so that we can get the right folks to take a look? FYI @davidmrdavid

BTW, Connor is no longer working on the project, but I think David should be able to help.

vany0114commented, Feb 8, 2022

@cgillum @ConnorMcMahon I’m experiencing this issue, I’m using the JS SDK to orchestrate a couple of timers, this error is causing the timer never to get fired off, any idea?


Thanks in advance.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Durable Functions Troubleshooting Guide - Azure
This Azure Storage messaging KQL query can help identify problems with dequeuing orchestration messages. If the problem impacts only a single ...
Read more >
Troubleshoot error: Azure Functions Runtime is unreachable
The most common reason for this is that the function app has lost access to its storage account. For more information, see Storage...
Read more >
Data persistence and serialization in Durable Functions
Messages are deleted after being processed, but instance states persist unless they're explicitly deleted by the application or an operator. In ...
Read more >
StorSimple 8000 series migration to Azure File Sync
Learn how to migrate a StorSimple 8100 or 8600 appliance to Azure File ... Phase 1: Prepare for migration; Phase 2: Deploy Azure...
Read more >
Disaster recovery - Azure Databricks
A clear disaster recovery pattern is critical for a cloud-native data analytics platform such as Azure Databricks.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found