question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] Memory usage problem using OpenReadAsync inside a list

See original GitHub issue

Describe the bug There is a memory issue when using OpenReadAsync inside a list. In our code, we get all the blobs from a container and add them to a list. We use the OpenReadAsyc. Once the list is created, we loop through all the items and add them to a zip file. With the WindowsAzure.Storage package 9.3.3 version, there were no problems, but with the Azure.Storage.Blobs package version 12.8.3 we have a memory leak problem. In each iteration, the memory goes up. When we have many blobs, the service crashes. With the previous version the memory usage is stable, but with Azure.Storage.Blobs is not.

Expected behavior What is the expected behavior? Memory usage should be stable, as it was in the WindowsAzure.Storage package. Memory usage in WindowsAzure.Storage:

image

Actual behavior (include Exception or Stack Trace) Memory usage using Azure.Storage.Blobs:

image

To Reproduce You can reproduce this behavior using this simple test to upload some blobs and then loop through them doing an OpenReadAsync.

WindowsAzure.Storage:

const string folderName = "test";

var blobClient = CloudStorageAccount.Parse("UseDevelopmentStorage=true").CreateCloudBlobClient();
var azureContainer = blobClient.GetContainerReference(_containerName);
await azureContainer.CreateIfNotExistsAsync();

for (var i = 0; i < 1000; i++)
{
    var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");

    var blob = azureContainer.GetBlockBlobReference(blobName);
    await blob.UploadFromByteArrayAsync(content, 0, content.Length).ConfigureAwait(false);
}

var blobList = new List<Task<Stream>>();
for (var i = 0; i < 1000; i++)
{
    var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");
    var blob = azureContainer.GetBlobReference(blobName);

    blobList.Add(blob.OpenReadAsync());
}

Azure.Storage.Blobs. The memory goes up on every iteration of the second foreach

var content = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 };
const string folderName = "test";

var blobClient = new BlobServiceClient("UseDevelopmentStorage=true");
var azureContainer = blobClient.GetBlobContainerClient(_containerName);
await azureContainer.CreateIfNotExistsAsync();

for (var i = 0; i < 1000; i++)
{
    var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");

    var blob = azureContainer.GetBlobClient(blobName);
    using (var stream = new MemoryStream(content, 0, content.Length))
    {
        await blob.UploadAsync(stream, overwrite: true).ConfigureAwait(false);
    }
}

var blobList = new List<Task<Stream>>();
for (var i = 0; i < 1000; i++)
{
    var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");
    var blob = azureContainer.GetBlobClient(blobName);

    blobList.Add(blob.OpenReadAsync());
}

Environment:

  • Name and version of the Library package used: Azure.Storage.Blobs 12.8.3
  • Hosting platform or OS and .NET runtime version: Azure AppService .NET Framework 4.6.2
  • IDE and version: Visual Studio 16.8.3

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
kasobol-msftcommented, Jun 23, 2021

Buffer size is in bytes.

1reaction
amishra-devcommented, Jun 21, 2021

@kasobol-msft can you look?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Azure BlobBaseClient.OpenReadAsync() leads to: System. ...
It looks to be a bug or so(created an issue) - when using OpenReadAsync() it adds 4 bytes on top of the stream,...
Read more >
[BUG] Blob Clientside Encryption OutOfMemoryException, ...
The problem only occurs during the upload/encryption part. How soon the error appears depends a bit on how much memory you give the...
Read more >
List entries / section leaks memory, says Instruments
List entries / section leaks memory, says Instruments ... Running the app just works fine but running it in Instruments results in a...
Read more >
How to use Azure Blob Storage with ASP.NET Core Web API
How to use Azure Blob Storage in an ASP.NET Core Web API to list, upload, download, and delete files. Learn how to make...
Read more >
Bug - Editor use too much memory resource.
I test build a game scene, the game player uses only 1GB of memory, very nice. But in the Editor, the Unity Editor...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found