[BUG] Blob Clientside Encryption OutOfMemoryException, ArgumentNullException (System.Buffers.TlsOverPerCoreLockedStacksArrayPool)
See original GitHub issueLibrary name and version
Azure.Storage.Blobs, 12.15.0; Azure.Identity, 1.8.2; Azure.Security.KeyVault.Keys, 4.5.0
Describe the bug
When uploading blobs with clientside encryption, and running the program in docker container with memory constraint, I am getting OutOfMemoryException and/or ArgumentNullException with stack traces like the below:
Unhandled exception. System.AggregateException: One or more errors occurred. (Value cannot be null. (Parameter 'array'))
---> System.ArgumentNullException: Value cannot be null. (Parameter 'array')
at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Return(T[] array, Boolean clearArray)
at Azure.Storage.Cryptography.AuthenticatedRegionCryptoStream.ReadInternal(Byte[] buffer, Int32 offset, Int32 count, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Cryptography.AuthenticatedRegionCryptoStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Azure.Storage.Shared.PooledMemoryStream.ReadLoopInternal(Stream stream, Byte[] buffer, Int32 offset, Int32 minCount, Int32 maxCount, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Shared.PooledMemoryStream.BufferStreamPartitionInternal(Stream stream, Int64 minCount, Int64 maxCount, Int64 absolutePosition, ArrayPool`1 arrayPool, Nullable`1 maxArrayPoolRentalSize, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.PartitionedUploader`2.UploadInternal(Stream content, Nullable`1 expectedContentLength, TServiceSpecificData args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClient.StagedUploadInternal(Stream content, BlobUploadOptions options, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)
at System.GC.<AllocateUninitializedArray>g__AllocateNewUninitializedArray|66_0[T](Int32 length, Boolean pinned)
at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)
at Azure.Storage.Cryptography.AuthenticatedRegionCryptoStream..ctor(Stream innerStream, IAuthenticatedCryptographicTransform transform, Int32 regionDataSize, CryptoStreamMode streamMode)
at Azure.Storage.Cryptography.ClientSideEncryptorV2_0.EncryptInternal(Stream plaintext, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClientSideEncryptor.ClientSideEncryptInternal(Stream content, IDictionary`2 metadata, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClient.StagedUploadInternal(Stream content, BlobUploadOptions options, Boolean async, CancellationToken cancellationToken)
The bigger the files and the smaller the memory the container has the sooner the exceptions are raised, e.g.:
documents around 2-5 Mb, memory limit 50Mb - after 10-20 seconds documents around 200-500 kb, memory limit 100Mb - after 2-3 minutes documents around 100k, memory limit 500Mb - 1x/day (in production)
Expected behavior
No OOM or other exceptions
Actual behavior
Exceptions are raised
Reproduction Steps
Azure setup:
- Set up a blob storage account with 2 containers -
source
anddestination
- Set up a Key vault key (RSA, 4096) for clientside encryption and set
ENCRYPTION_MASTERKEY
env var - Configure app registration/service principal with permission to access the key vault, and set
AZURE_CLIENT_ID
,AZURE_CLIENT_SECRET
andAZURE_TENANT_ID
env vars - Upload a few large files (2-5 Mb each) to the source container in the blob storage account
Application:
<PackageReference Include="FSharp.Core" />
<PackageReference Include="Azure.Storage.Blobs" />
<PackageReference Include="Azure.Identity" />
<PackageReference Include="Azure.Security.KeyVault.Keys" />
module BlobOOMRepro.Program
open System
open System.Diagnostics
open Azure.Core
open Azure.Core.Cryptography
open Azure.Identity
open Azure.Security.KeyVault.Keys.Cryptography
open Azure.Storage
open Azure.Storage.Blobs
open Azure.Storage.Blobs.Specialized
module AzureIdentity =
let getTokenCredential () =
if
not (
String.IsNullOrEmpty(
Environment.GetEnvironmentVariable("AZURE_CLIENT_ID", EnvironmentVariableTarget.Process)
)
)
&& not (
String.IsNullOrEmpty(
Environment.GetEnvironmentVariable("AZURE_TENANT_ID", EnvironmentVariableTarget.Process)
)
)
&& not (
String.IsNullOrEmpty(
Environment.GetEnvironmentVariable("AZURE_CLIENT_SECRET", EnvironmentVariableTarget.Process)
)
)
then
EnvironmentCredential() :> TokenCredential
else
failwith "AZURE_CLIENT_ID, AZURE_TENANT_ID or AZURE_CLIENT_SECRET env var not found"
module AzureKeyVault =
let getCryptoClient (keyId: Uri) : CryptographyClient =
CryptographyClient(keyId, AzureIdentity.getTokenCredential ())
module AzureBlobStorage =
let getEncryptionOptions (keyUri: string) : SpecializedBlobClientOptions =
// Key and key resolver instances through KeyVault SDK
let keyEncryptionKey =
AzureKeyVault.getCryptoClient (Uri keyUri) :> IKeyEncryptionKey
let keyResolver = KeyResolver(AzureIdentity.getTokenCredential ())
// Create the encryption options to be used for upload and download.
let encryptionOptions =
ClientSideEncryptionOptions(ClientSideEncryptionVersion.V2_0)
encryptionOptions.KeyEncryptionKey <- keyEncryptionKey
encryptionOptions.KeyResolver <- keyResolver
encryptionOptions.KeyWrapAlgorithm <- "RSA1_5" // string the storage client will use when calling IKeyEncryptionKey.WrapKey()
// Set the encryption options on the client options
let options = SpecializedBlobClientOptions() //:> BlobClientOptions
options.ClientSideEncryption <- encryptionOptions
options
let encryptionOptions =
AzureBlobStorage.getEncryptionOptions (Environment.GetEnvironmentVariable("ENCRYPTION_MASTERKEY"))
[<EntryPoint>]
let main argv =
printfn "Press Enter"
Console.ReadLine() |> ignore
let sw = Stopwatch()
sw.Start()
let files =
[|
"https://SOMEACCOUNT.blob.core.windows.net/source/large_image_1.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_1.jpg" // 8,5MB
"https://SOMEACCOUNT.blob.core.windows.net/source/large_image_2.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_2.jpg" // 2,8MB
"https://SOMEACCOUNT.blob.core.windows.net/source/large_image_3.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_3.jpg" // 3,6MB
"https://SOMEACCOUNT.blob.core.windows.net/source/large_image_4.jpg", "https://SOMEACCOUNT.blob.core.windows.net/destination/large_image_4.jpg" // 2,8MB
|]
[ 1..100 ]
|> List.iter (fun c ->
async {
let (srcBlobUri, destBlobUri) = files[c % files.Length]
let srcBlobClient = BlobClient(srcBlobUri |> Uri, AzureIdentity.getTokenCredential())
let destBlobClient = BlobClient(destBlobUri |> Uri, AzureIdentity.getTokenCredential(), encryptionOptions)
use! srcBlobStream = srcBlobClient.OpenReadAsync() |> Async.AwaitTask
srcBlobStream.Position <- 0 // reset stream so that it can be read again
do!
destBlobClient.UploadAsync(srcBlobStream, true)
|> Async.AwaitTask
|> Async.Ignore
printfn $"{srcBlobUri} -> {destBlobUri}"
} |> Async.RunSynchronously
)
printfn "Done"
sw.Stop()
printfn $"Elapsed = %A{sw.Elapsed}"
0 // return an integer exit code
build and run in docker container:
dotnet clean -c Release -o publish
dotnet publish -c Release -o publish
docker build -t DOESNOTMATTER.azurecr.io/blob-oom-repro:latest .
docker run --name blob-oom-repro --add-host=host.docker.internal:172.17.0.1 -p 7182:80 -m 50m -e AZURE_TENANT_ID -e AZURE_CLIENT_ID -e AZURE_CLIENT_SECRET -i -e ENCRYPTION_MASTERKEY --rm DOESNOTMATTER.azurecr.io/blob-oom-repro:latest
Environment
.NET SDK: Version: 7.0.202 Commit: 6c74320bc3
Runtime Environment: OS Name: ubuntu OS Version: 22.04 OS Platform: Linux RID: ubuntu.22.04-x64 Base Path: /usr/share/dotnet/sdk/7.0.202/
Host: Version: 7.0.4 Architecture: x64 Commit: 0a396acafe
.NET SDKs installed: 7.0.202 [/usr/share/dotnet/sdk]
.NET runtimes installed: Microsoft.AspNetCore.App 7.0.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 7.0.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Issue Analytics
- State:
- Created 6 months ago
- Comments:13 (7 by maintainers)
Top GitHub Comments
I’ve submitted #35247 to patch a verified memory leak which I believe you are encountering in your scenario.
Looks like the above PR linked resolved the issue. Closing the issue.