[BUG]BlobClient.DownloadToAsync() does not work consistently in .Net 6.

Describe the bug I spent time today updating a small web app from .Net 5.0 to .Net 6.0. After the upgrade, downloading small blob files (just a few kb) will fail sporadically. Our app calls BlobClient.DownloadToAsync(Stream, CancellationToken).

This app has run on .Net 5.0 for over 6 months and Net Core 3.1 for at least a year before that. We’ve never had an issue downloading blobs.

Sometimes the download succeeds, and sometimes it fails. When failing, I’m experiencing a variety of failures…always a problem processing the http headers.

Actual behavior (include Exception or Stack Trace) What is the actual behavior?

Here are a couple of different errors we’ve received. Both are related to processing the http headers.

Error 1: Index was outside the bounds of the array.

at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex)

   at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue)

   at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext()

   at Azure.Core.Pipeline.HttpClientTransport.GetHeaders(HttpHeaders headers, HttpContent content)+MoveNext()

   at Azure.Core.Pipeline.LoggingPolicy.FormatHeaders(IEnumerable`1 headers)

   at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>g__ProcessAsyncInner|4_0(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Storage.Blobs.BlobRestClient.DownloadAsync(String snapshot, String versionId, Nullable`1 timeout, String range, String leaseId, Nullable`1 rangeGetContentMD5, Nullable`1 rangeGetContentCRC64, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartDownloadAsync(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, Int64 startOffset, Boolean async, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadStreamingInternal(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, String operationName, Boolean async, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadStreamingAsync(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.PartitionedDownloader.DownloadToAsync(Stream destination, BlobRequestConditions conditions, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.StagedDownloadAsync(Stream destination, BlobRequestConditions conditions, StorageTransferOptions transferOptions, Boolean async, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadToAsync(Stream destination, BlobRequestConditions conditions, StorageTransferOptions transferOptions, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadToAsync(Stream destination, CancellationToken cancellationToken)

Error 2: Collection was modified; enumeration operation may not execute.

at System.Collections.Generic.List`1.Enumerator.MoveNext()

   at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex)

   at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue)

   at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext()

   at Azure.Core.Pipeline.HttpClientTransport.GetHeaders(HttpHeaders headers, HttpContent content)+MoveNext()

   at Azure.Core.Pipeline.LoggingPolicy.FormatHeaders(IEnumerable`1 headers)

   at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>g__ProcessAsyncInner|4_0(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)

   at Azure.Storage.Blobs.BlobRestClient.GetPropertiesAsync(String snapshot, String versionId, Nullable`1 timeout, String leaseId, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesInternal(BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken, String operationName)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.ExistsInternal(Boolean async, CancellationToken cancellationToken)

   at Azure.Storage.Blobs.Specialized.BlobBaseClient.ExistsAsync(CancellationToken cancellationToken)

To Reproduce Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

Here’s the code snippet that is attempting to download the blob. I’ve also included the Startup code for injecting the BlobServiceClient.

            services
                .AddAzureClients(builder =>
                {
                    builder
                        .AddBlobServiceClient(Configuration.GetConnectionString("BlobStorageConnection"));
                });

            using var destination = new MemoryStream();

            var containerClient = _client.GetBlobContainerClient(containerName.ToLower());
            var blobClient = containerClient.GetBlobClient(fileName.ToLower());
            if (!(await blobClient.ExistsAsync(cancellationToken).ConfigureAwait(false)).Value)
                throw new FileNotFoundException($"The specified blob for this attachment does not exist.  Container: {attachment.ContainerName}, File Name: {attachment.FileName}.");

            await blobClient.DownloadToAsync(
                destination: destination,
                cancellationToken: cancellationToken).ConfigureAwait(false);

Environment:

• Name and version of the Library package used: [e.g. Azure.Storage.Blobs 12.9.0]
• Windows Server 2019. Running .Net 6.0.0. (6.0.0-rtm.21522.10)
• Web App running in IIS in Inprocess mode.

[brianjscho]

I worked on this weekend. Since I had to install this tool, I thought I’d spin up a new VM at azure to test with a completely clean slate. I was NOT able to reproduce the issue. That’s when it clicked. The link above mentions that New Relic v 9.2.0 contained a fix for this issue. We use New Relic to monitor our applications and are only running v9.0 in all our environments. I’ve updated the VMs in 2 of our non-production environments to the latest version of their monitoring agent (v9.4) and the issue is resolved for these VMs.

My apologies for not mentioning New Relic in the first place. It never occurred to me that New Relic might be the issue. I think we can probably close this issue unless you feel there is still something to dig into further.

[christothes]

My apologies for not mentioning New Relic in the first place. It never occurred to me that New Relic might be the issue. I think we can probably close this issue unless you feel there is still something to dig into further.

No worries. I don’t think we connected the dots until @adityamandaleeka referenced the runtime issue either.

Thanks for confirming!