DurableEntityClient cannot retrieve large message from blob storage

Description

We have followed the official documentation to setup our durable functions for zero-downtime deployments (see: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-zero-downtime-deployment)

With this scenario we have 3 different storage accounts configured:

1. StorageAccount1 (shared between our 2 deployment slots)

• Instance and History tables are located in this Storage Account
• This Storage Account is referenced as the “AzureWebJobStorage” in the application settings of both slots

1. StorageAccount2 (used by production slot)

• WorkItem and Control Queues for production slot are located in this Storage Account
• this is referenced as the “DurableManagementStorage” in the application settings of the production slot

1. StorageAccount3 (used by CD slot)

• WorkItem and Control Queues for CD slot are located in this Storage Account
• this is referenced as the “DurableManagementStorage” in the application settings of the CD slot

Within our code (which is not always running in the context of a durable function), we are retrieving the state of Durable Entities using DurableClientFactory to create a DurableEntityClient by specifiying the following DurableClientOptions:

• ConnectionName -> set to AzureWebJobStorage (pointing to StorageAccount1)
• TaskHubName -> set to the same TaskHub used by the durable function
• IsEcternalClient -> true

In some rare scenarios, our Durable Entity State might contain more than 16KB of data, in which case the content gets written to a blob. The strange thing is that this blob container will be located in StorageAccount2 or StorageAccount3 and not in StorageAccount1 as we would expect. Once a large message gets written to a blob and we then try to retrieve the Entity State using a DurableEntityClient, we get the below error:

DurableTask.AzureStorage.Storage.DurableTaskStorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. —> Microsoft.WindowsAzure.Storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable1 offset, Nullable1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress1 progressHandler, CancellationToken cancellationToken) at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 156 at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId) at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 136 Request Information RequestID:99d56c46-601e-005e-2daf-554c71000000 RequestDate:Mon, 13 Mar 2023 13:25:32 GMT StatusMessage:Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode:AuthenticationFailed ErrorMessage:Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:99d56c46-601e-005e-2daf-554c71000000 Time:2023-03-13T13:25:32.0168802Z

If we use a single Storage Account (without specifying a separate storage account for DurableManagementStorage) everything works correctly, so the issue seems not to be on our end.

Expected behavior

It should be possible to retrieve Durable Entity State by using a DurableEntityClient even if separate storage accounts are configured for DurableManagementStorage and AzureWebJobStorage

App Details

• Durable Functions extension version (e.g. v1.8.3): 2.9.2
• Azure Functions runtime version (1.0 or 2.0): 4
• Programming language used: C#

[sebastianburckhardt]

In my understanding this is a bug in the Azure Storage provider and I have filed an issue azure/durabletask#879.

[nicolaor]

Hi @sebastianburckhardt

The problem with accessing the blob, is that an external client application (in our case this is a web application which visualizes the state of our durable entities) is only aware of the shared storage (the one containing the tables). The storage account(s) where the large messages get stored are completely unknown to that client.

In the external client application, we use the following approach to access durable entity state:

• We added the DurableClientFactory in startup -> builder.Services.AddDurableClientFactory();
• We then create a DurableEntityClient from that factory by using the following DurableClientOptions: – ConnectionName: reference to the connection string pointing to the shared storage account (where tables are located) – TaskHub: the taskhub name (same as the one configured for the durable functions) – IsExternalClient: true

We are not accessing the storage with the managed identity here, but I don’t think this is relevant. With this configuration, the external client is able to retrieve the entity states, it only fails once a large message is generated.