[BUG] HttpHeaders+HeaderStoreItemInfo stay in memory from fetching BlobHierarchyItem objects by GetBlobsByHierarchyAsync

Describe the bug

Increase of HttpHeaders objects relating to BlobHierarchyItem objects when calling GetBlobsByHierarchyAsync.

Expected behavior No increase in objects after each run.

Actual behavior (include Exception or Stack Trace) Increase in objects, structurally about 10k HttpHeaders each run with no sign of garbage collection.

To Reproduce

1. Make a BlobContainerClient
2. Create some blobs that are multiple levels deep, in my case I have about 10 level deep blob storage structure, but I start my 7 levels deep for instance. Then I iterate down from level 7 to level 10.
3. Create some blob storage prefix values.
4. Iterate trough the prefixes and multiple containers.
5. With method:

 public async Task GetBlobNamesByHierarchy(BlobContainerClient blobContainerClient, string prefix, ConcurrentBag<string> blobList)
        {
            await foreach (Page<BlobHierarchyItem> page in blobContainerClient.GetBlobsByHierarchyAsync(BlobTraits.None, BlobStates.None, "/", prefix).AsPages())
            {
                var prefixes = page.Values.Where(item => item.IsPrefix).Select(item => item.Prefix);

                foreach (var blob in page.Values.Where(item => item.IsBlob).Select(item => item.Blob))
                {
                    blobList.Add(blob.Name);
                }

                foreach (var recursedPrefix in prefixes)
                {
                    await GetBlobNamesByHierarchy(blobContainerClient, recursedPrefix, blobList);
                }
            }
        }

Environment:

• Azure.Storage.Blobs 12.9…0
• .NET 3.1 Function App project inside VS 2019 version 16.10.1

.NET SDK (reflecting any global.json):
 Version:   5.0.301
 Commit:    ef17233f86

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.301\

Host (useful for support):
  Version: 5.0.7
  Commit:  556582d964

.NET SDKs installed:
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.509 [C:\Program Files\dotnet\sdk]
  3.1.108 [C:\Program Files\dotnet\sdk]
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.405 [C:\Program Files\dotnet\sdk]
  5.0.101 [C:\Program Files\dotnet\sdk]
  5.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[tg-msft]

I’d like to look into this on the Azure.Core side because there shouldn’t be a difference in allocations between any of our sync vs. async methods. Reopening and removing Storage as the owner.

[erwinkramer]

After some testing I found a workaround. When using the synchronous GetBlobsByHierarchy method, no increase in HttpHeaders objects is observed.