Blob upload results in excessive memory usage/leak

Which service(blob, file, queue, table) does this issue concern?

Blob

Which version of the SDK was used?

Microsoft.Azure.Storage.Blob -Version 10.0.3

Which platform are you using? (ex: .NET Core 2.1)

.net core 2.2.300

What problem was encountered?

OutOfMemory / excessive memory usage

How can we reproduce the problem in the simplest way?

namespace ConsoleAppAzureBlobMemoryIssue
{
    class Program
    {
        static void Main(string[] args)
        {
            CloudStorageAccount account = CloudStorageAccount.Parse("StorageConnectionString");
            CloudBlobClient serviceClient = account.CreateCloudBlobClient();

            var container = serviceClient.GetContainerReference("backup");
            container.CreateIfNotExistsAsync().Wait();

            string path = @"c:\folder\gigafile.zip"; 
            CloudBlockBlob blob = container.GetBlockBlobReference(Path.GetFileName(path));
            blob.UploadFromFileAsync(path).Wait();
        }
    }
}

Run the program, open Task Manager and “enjoy” how the memory usage of dotnet.exe explodes as the file gets uploaded, the memory usages increases directly with amount of bytes uploaded sofar. So if you attempt to upload a 50GB file you will need +50GB of free memory.

Have you found a mitigation/solution?

Last working version is WindowsAzure.Storage -Version 9.3.3, newer versions contains the issue.

[markheath]

I’d be interested to know if there are any plans to fix this issue soon, or whether there is an underlying technical limitation that requires the entire file to be held in memory. I recently updated our application to the latest version of Microsoft.Azure.Storage.Blob following the advice given here, but this high memory usage is an absolute showstopper for us, and looks like I’m going to have to roll everything back to WindowsAzure.Storage 9.3.3.

Interestingly it doesn’t seem to be a problem if the source isn’t from a file. I can use UploadFromStreamAsync to copy from a stream from another large blob opened with OpenReadAsync and memory usage stays well below the overall blob size.

[seanmcc-msft]

Hi @GFoley83 , if I recall correctly, it was fixed in both.

-Sean