question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] Blob Clientside Encryption OutOfMemoryException, ArgumentNullException (System.Buffers.TlsOverPerCoreLockedStacksArrayPool)

See original GitHub issue

Library name and version

Azure.Storage.Blobs, 12.15.0; Azure.Identity, 1.8.2; Azure.Security.KeyVault.Keys, 4.5.0

Describe the bug

When uploading blobs with clientside encryption, and running the program in docker container with memory constraint, I am getting OutOfMemoryException and/or ArgumentNullException with stack traces like the below:

Unhandled exception. System.AggregateException: One or more errors occurred. (Value cannot be null. (Parameter 'array'))
 ---> System.ArgumentNullException: Value cannot be null. (Parameter 'array')
   at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Return(T[] array, Boolean clearArray)
   at Azure.Storage.Cryptography.AuthenticatedRegionCryptoStream.ReadInternal(Byte[] buffer, Int32 offset, Int32 count, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Cryptography.AuthenticatedRegionCryptoStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Azure.Storage.Shared.PooledMemoryStream.ReadLoopInternal(Stream stream, Byte[] buffer, Int32 offset, Int32 minCount, Int32 maxCount, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Shared.PooledMemoryStream.BufferStreamPartitionInternal(Stream stream, Int64 minCount, Int64 maxCount, Int64 absolutePosition, ArrayPool`1 arrayPool, Nullable`1 maxArrayPoolRentalSize, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.PartitionedUploader`2.UploadInternal(Stream content, Nullable`1 expectedContentLength, TServiceSpecificData args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobClient.StagedUploadInternal(Stream content, BlobUploadOptions options, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)
at System.GC.<AllocateUninitializedArray>g__AllocateNewUninitializedArray|66_0[T](Int32 length, Boolean pinned)
at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)
at Azure.Storage.Cryptography.AuthenticatedRegionCryptoStream..ctor(Stream innerStream, IAuthenticatedCryptographicTransform transform, Int32 regionDataSize, CryptoStreamMode streamMode)
at Azure.Storage.Cryptography.ClientSideEncryptorV2_0.EncryptInternal(Stream plaintext, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClientSideEncryptor.ClientSideEncryptInternal(Stream content, IDictionary`2 metadata, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClient.StagedUploadInternal(Stream content, BlobUploadOptions options, Boolean async, CancellationToken cancellationToken)

The bigger the files and the smaller the memory the container has the sooner the exceptions are raised, e.g.:

documents around 2-5 Mb, memory limit 50Mb - after 10-20 seconds documents around 200-500 kb, memory limit 100Mb - after 2-3 minutes documents around 100k, memory limit 500Mb - 1x/day (in production)

Expected behavior

No OOM or other exceptions

Actual behavior

Exceptions are raised

Reproduction Steps

Azure setup:

  1. Set up a blob storage account with 2 containers - source and destination
  2. Set up a Key vault key (RSA, 4096) for clientside encryption and set ENCRYPTION_MASTERKEY env var
  3. Configure app registration/service principal with permission to access the key vault, and set AZURE_CLIENT_ID, AZURE_CLIENT_SECRET and AZURE_TENANT_ID env vars
  4. Upload a few large files (2-5 Mb each) to the source container in the blob storage account

Application:

        <PackageReference Include="FSharp.Core" />
        <PackageReference Include="Azure.Storage.Blobs" />
        <PackageReference Include="Azure.Identity" />
        <PackageReference Include="Azure.Security.KeyVault.Keys" />

module BlobOOMRepro.Program

open System
open System.Diagnostics
open Azure.Core
open Azure.Core.Cryptography
open Azure.Identity
open Azure.Security.KeyVault.Keys.Cryptography
open Azure.Storage
open Azure.Storage.Blobs
open Azure.Storage.Blobs.Specialized

module AzureIdentity =

    let getTokenCredential () =
        if
            not (
                String.IsNullOrEmpty(
                    Environment.GetEnvironmentVariable("AZURE_CLIENT_ID", EnvironmentVariableTarget.Process)
                )
            )
            && not (
                String.IsNullOrEmpty(
                    Environment.GetEnvironmentVariable("AZURE_TENANT_ID", EnvironmentVariableTarget.Process)
                )
            )
            && not (
                String.IsNullOrEmpty(
                    Environment.GetEnvironmentVariable("AZURE_CLIENT_SECRET", EnvironmentVariableTarget.Process)
                )
            )
        then
            EnvironmentCredential() :> TokenCredential
        else
            failwith "AZURE_CLIENT_ID, AZURE_TENANT_ID or AZURE_CLIENT_SECRET env var not found"

module AzureKeyVault =

    let getCryptoClient (keyId: Uri) : CryptographyClient =
        CryptographyClient(keyId, AzureIdentity.getTokenCredential ())

module AzureBlobStorage =

    let getEncryptionOptions (keyUri: string) : SpecializedBlobClientOptions =
        // Key and key resolver instances through KeyVault SDK
        let keyEncryptionKey =
            AzureKeyVault.getCryptoClient (Uri keyUri) :> IKeyEncryptionKey

        let keyResolver = KeyResolver(AzureIdentity.getTokenCredential ())

        // Create the encryption options to be used for upload and download.
        let encryptionOptions =
            ClientSideEncryptionOptions(ClientSideEncryptionVersion.V2_0)

        encryptionOptions.KeyEncryptionKey <- keyEncryptionKey
        encryptionOptions.KeyResolver <- keyResolver
        encryptionOptions.KeyWrapAlgorithm <- "RSA1_5" // string the storage client will use when calling IKeyEncryptionKey.WrapKey()

        // Set the encryption options on the client options
        let options = SpecializedBlobClientOptions() //:> BlobClientOptions
        options.ClientSideEncryption <- encryptionOptions

        options

let encryptionOptions =
    AzureBlobStorage.getEncryptionOptions (Environment.GetEnvironmentVariable("ENCRYPTION_MASTERKEY"))

[<EntryPoint>]
let main argv =
    printfn "Press Enter"
    Console.ReadLine() |> ignore

    let sw = Stopwatch()
    sw.Start()

    let files =
        [|
            "https://SOMEACCOUNT.blob.core.windows.net/source/large_image_1.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_1.jpg" // 8,5MB
            "https://SOMEACCOUNT.blob.core.windows.net/source/large_image_2.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_2.jpg" // 2,8MB
            "https://SOMEACCOUNT.blob.core.windows.net/source/large_image_3.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_3.jpg" // 3,6MB
            "https://SOMEACCOUNT.blob.core.windows.net/source/large_image_4.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_4.jpg" // 2,8MB
        |]

    [ 1..100 ]
    |> List.iter (fun c ->
        async {
        let (srcBlobUri, destBlobUri) = files[c % files.Length]
        
        let srcBlobClient = BlobClient(srcBlobUri |> Uri, AzureIdentity.getTokenCredential())
        
        let destBlobClient = BlobClient(destBlobUri |> Uri, AzureIdentity.getTokenCredential(), encryptionOptions)
        
        use! srcBlobStream = srcBlobClient.OpenReadAsync() |> Async.AwaitTask
        
        srcBlobStream.Position <- 0 // reset stream so that it can be read again

        do!
            destBlobClient.UploadAsync(srcBlobStream, true)
                |> Async.AwaitTask
                |> Async.Ignore
            
        printfn $"{srcBlobUri} -> {destBlobUri}"

        } |> Async.RunSynchronously
    )

    printfn "Done"

    sw.Stop()

    printfn $"Elapsed = %A{sw.Elapsed}"

    0 // return an integer exit code

build and run in docker container:

dotnet clean -c Release -o publish 
dotnet publish -c Release -o publish 
docker build -t DOESNOTMATTER.azurecr.io/blob-oom-repro:latest .

docker run --name blob-oom-repro --add-host=host.docker.internal:172.17.0.1 -p 7182:80 -m 50m -e AZURE_TENANT_ID -e AZURE_CLIENT_ID -e AZURE_CLIENT_SECRET -i -e ENCRYPTION_MASTERKEY --rm DOESNOTMATTER.azurecr.io/blob-oom-repro:latest

Environment

.NET SDK: Version: 7.0.202 Commit: 6c74320bc3

Runtime Environment: OS Name: ubuntu OS Version: 22.04 OS Platform: Linux RID: ubuntu.22.04-x64 Base Path: /usr/share/dotnet/sdk/7.0.202/

Host: Version: 7.0.4 Architecture: x64 Commit: 0a396acafe

.NET SDKs installed: 7.0.202 [/usr/share/dotnet/sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 7.0.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 7.0.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Issue Analytics

  • State:closed
  • Created 6 months ago
  • Comments:13 (7 by maintainers)

github_iconTop GitHub Comments

2reactions
jaschrep-msftcommented, Mar 30, 2023

I’ve submitted #35247 to patch a verified memory leak which I believe you are encountering in your scenario.

0reactions
amnguyecommented, Jun 9, 2023

Looks like the above PR linked resolved the issue. Closing the issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

https://raw.githubusercontent.com/dotnet/samples/m...
OutOfMemoryException""" Opened on behalf of @Jiayili1 The test `System. ... shared/System/Buffers/TlsOverPerCoreLockedStacksArrayPool.cs:line 178 at System.
Read more >
Client-side encryption for blobs - Azure Storage
The Blob Storage client library supports client-side encryption and integration with Azure Key Vault for users requiring encryption on the ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found