[BUG] Memory usage problem using OpenReadAsync inside a list
See original GitHub issueDescribe the bug There is a memory issue when using OpenReadAsync inside a list. In our code, we get all the blobs from a container and add them to a list. We use the OpenReadAsyc. Once the list is created, we loop through all the items and add them to a zip file. With the WindowsAzure.Storage package 9.3.3 version, there were no problems, but with the Azure.Storage.Blobs package version 12.8.3 we have a memory leak problem. In each iteration, the memory goes up. When we have many blobs, the service crashes. With the previous version the memory usage is stable, but with Azure.Storage.Blobs is not.
Expected behavior What is the expected behavior? Memory usage should be stable, as it was in the WindowsAzure.Storage package. Memory usage in WindowsAzure.Storage:
Actual behavior (include Exception or Stack Trace) Memory usage using Azure.Storage.Blobs:
To Reproduce You can reproduce this behavior using this simple test to upload some blobs and then loop through them doing an OpenReadAsync.
WindowsAzure.Storage:
const string folderName = "test";
var blobClient = CloudStorageAccount.Parse("UseDevelopmentStorage=true").CreateCloudBlobClient();
var azureContainer = blobClient.GetContainerReference(_containerName);
await azureContainer.CreateIfNotExistsAsync();
for (var i = 0; i < 1000; i++)
{
var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");
var blob = azureContainer.GetBlockBlobReference(blobName);
await blob.UploadFromByteArrayAsync(content, 0, content.Length).ConfigureAwait(false);
}
var blobList = new List<Task<Stream>>();
for (var i = 0; i < 1000; i++)
{
var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");
var blob = azureContainer.GetBlobReference(blobName);
blobList.Add(blob.OpenReadAsync());
}
Azure.Storage.Blobs. The memory goes up on every iteration of the second foreach
var content = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 };
const string folderName = "test";
var blobClient = new BlobServiceClient("UseDevelopmentStorage=true");
var azureContainer = blobClient.GetBlobContainerClient(_containerName);
await azureContainer.CreateIfNotExistsAsync();
for (var i = 0; i < 1000; i++)
{
var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");
var blob = azureContainer.GetBlobClient(blobName);
using (var stream = new MemoryStream(content, 0, content.Length))
{
await blob.UploadAsync(stream, overwrite: true).ConfigureAwait(false);
}
}
var blobList = new List<Task<Stream>>();
for (var i = 0; i < 1000; i++)
{
var blobName = BlobPath.Combine(folderName, $"{_blobName}{i}");
var blob = azureContainer.GetBlobClient(blobName);
blobList.Add(blob.OpenReadAsync());
}
Environment:
- Name and version of the Library package used: Azure.Storage.Blobs 12.8.3
- Hosting platform or OS and .NET runtime version: Azure AppService .NET Framework 4.6.2
- IDE and version: Visual Studio 16.8.3
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (3 by maintainers)
Top GitHub Comments
Buffer size is in bytes.
@kasobol-msft can you look?