Eternal orchestrations with large messages have unbounded blob counts

Eternal orchestrations that use ContinueAsNew and have state stored in large messages will see their blob storage usage grow unbounded over time.

As of this PR, we no longer have random blob names, which means we can overwrite blobs from previous generations, reducing the impact of the problem. However, it does not handle message blobs, which will still continue to grow in an unbounded way since the blob names contain the sequence number, which does not reset. Fixing the message blob problem may be as simple as deleting message blobs at the same time we delete their corresponding queue messages (which we might want to do anyways).

[dsm0880]

@cgillum That makes sense, thank you for the quick response. I ask because we are experiencing a sudden increase in blob requests which is correlating to memory issues. The largemessages blob has ~63,000 messages in there as we do have a number of eternal orchestrations. I’ve seen the other ticket #340 which speaks of a memory leak which may very well be the case, but I think what I’m seeing is increased requests to blob because of the need to page the blob storage. We must have passed a threshold that caused the issue.

Regardless, as this ticket mentions the fix is to purge the largemessages. In the meantime, do you see anything wrong with simply deleting old large-messages for eternal orchestrations as a workaround?

Last, awesome 1.7 release. You guys rocked and totally fix issue #462.

[dsm0880]

@cgillum Thanks for the quick response. Let me revisit point 1 with my team and confirm. I’ll discuss point 2 with my team as well and see if we have any ideas that would be worth sharing here. Have a great weekend!