Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] occasional bad requests comming from BlobClient.UploadAsync

See original GitHub issue

Library name and version

Azure.Storage.Blobs 12.12.0

Describe the bug

I’ve upgraded from ASP.NET MVC 5 / WebAPI 2 using WindowsAzure.Storage to ASP.NET Core 6 using Azure.Storage.Blobs and after deploying it to our UAT System, immediately but inconsistently, we’ve started seeing Bad Request exceptions comming from the BlobClient.UploadAsync API.

Expected behavior

Uploading Blobs via the Client always uses the right request verb.

Actual behavior

Azure.RequestFailedException: Service request failed.
Status: 400 (Bad Request)

Content:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request - Invalid Verb</h2>
<hr><p>HTTP Error 400. The request verb is invalid.</p>
</BODY></HTML>


Headers:
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 05 May 2022 13:59:33 GMT
Connection: close
Content-Type: text/html; charset=us-ascii
Content-Length: 326

   at async Task<ResponseWithHeaders<BlockBlobUploadHeaders>> Azure.Storage.Blobs.BlockBlobRestClient.UploadAsync(long contentLength, Stream body, int? timeout, byte[] transactionalContentMD5, string blobContentType, string blobContentEncoding, string blobContentLanguage, byte[] blobContentMD5, string blobCacheControl, IDictionary<string, string> metadata, string leaseId, string blobContentDisposition, string encryptionKey, string encryptionKeySha256, EncryptionAlgorithmTypeInternal? encryptionAlgorithm, string encryptionScope, AccessTier? tier, DateTimeOffset? ifModifiedSince, DateTimeOffset? ifUnmodifiedSince, string ifMatch, string ifNoneMatch, string ifTags, string blobTagsString, DateTimeOffset? immutabilityPolicyExpiry, BlobImmutabilityPolicyMode? immutabilityPolicyMode, bool? legalHold, CancellationToken cancellationToken)
   at async Task<Response<BlobContentInfo>> Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadInternal(Stream content, BlobHttpHeaders blobHttpHeaders, IDictionary<string, string> metadata, IDictionary<string, string> tags, BlobRequestConditions conditions, AccessTier? accessTier, BlobImmutabilityPolicy immutabilityPolicy, bool? legalHold, IProgress<long> progressHandler, string operationName, bool async, CancellationToken cancellationToken)
   at async Behaviors Azure.Storage.Blobs.Specialized.BlockBlobClient.GetPartitionedUploaderBehaviors(BlockBlobClient client)+(?) => { } [0]
   at async Task<Response<TCompleteUploadReturn>> Azure.Storage.PartitionedUploader<TServiceSpecificData, TCompleteUploadReturn>.UploadInternal(Stream content, long? expectedContentLength, TServiceSpecificData args, IProgress<long> progressHandler, bool async, CancellationToken cancellationToken)
   at async Task<Response<BlobContentInfo>> Azure.Storage.Blobs.BlobClient.StagedUploadInternal(Stream content, BlobUploadOptions options, bool async, CancellationToken cancellationToken)

Reproduction Steps

var created = DateTime.UtcNow; // the real code gets created from a database which ensures it's unique (via unique index).
var container = new BlobContainerClient(AzureStorageConnectionString, "imports-2022-05");
var blob = container.GetBlobClient($"12345/{created:o}");
await blob.UploadAsync(stream, new BlobHttpHeaders { ContentType = "application/json" });

Environment

ASP.NET Core 6 InProcess app running in Azure App Service using an StorageV2 account in Germany West Central.

Issue Analytics

State:
Created a year ago
Comments:10 (8 by maintainers)

Top GitHub Comments

1reaction

navba-MSFTcommented, May 11, 2022

@Suchiman Thanks for getting back. I am adding the Service Team to look into this further.

@xgithubtriage Could you please look into this issue on priority and provide an update on this ? Awaiting your reply.

0reactions

Suchimancommented, Sep 12, 2022

The WindowsAzure.Storage library has been working flawlessly so far the past few months but we would still prefer to upgrade to this newer library. We’ve experimented with it again and quickly ran into this issue again with version 12.13.1. It seems that adding a retry policy is able to workaround the problem:

class RetryBadRequestPolicy : HttpPipelinePolicy
{
    private static readonly ILogger Logger = Log.ForContext<RetryBadRequestPolicy>();

    public override void Process(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline)
    {
        ProcessAsync(message, pipeline, false).GetAwaiter().GetResult();
    }

    public override ValueTask ProcessAsync(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline)
    {
        return ProcessAsync(message, pipeline, true);
    }

    private async ValueTask ProcessAsync(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async)
    {
        int attempt = 0;
        while (true)
        {
            if (attempt > 0)
            {
                if (message.HasResponse)
                {
                    // Dispose the content stream to free up a connection if the request has any
                    message.Response.ContentStream?.Dispose();
                }

                if (async)
                {
                    await Task.Delay(1000);
                }
                else
                {
                    Thread.Sleep(1000);
                }
            }

            try
            {
                if (async)
                {
                    await ProcessNextAsync(message, pipeline).ConfigureAwait(false);
                }
                else
                {
                    ProcessNext(message, pipeline);
                }
            }
            catch (RequestFailedException ex) when (ex.Status == 400 && attempt++ <= 2)
            {
                Logger.Warning(ex, "Retrying BadRequest Blob Storage Request");
                continue;
            }

            if (message.Response.Status != 400 || attempt++ > 2)
            {
                return;
            }
            Logger.Warning("Retrying BadRequest Blob Storage Request");
        }
    }
}

Top Results From Across the Web

[BUG] BlobClient.UploadAsync never completes #12811

Describe the bug BlobClient.UploadAsync never completes if it is called with a Stream where Position isn't 0. Expected behavior If should ...

Very occasional 400 Bad Request error when posting to ...

The error is transient - if I refresh the page on which it shows up, the object (with the same container and file...

BlobClient.UploadAsync Method (Azure.Storage.Blobs)

The UploadAsync(Stream, Boolean, CancellationToken) operation creates a new block blob or throws if the blob already exists. Setting overwrite to true allows ...

Azure Blob Storage error codes

Error code HTTP status code User message BlobAlreadyExists Conflict (409) The specified blob already exists. BlobNotFound Not Found (404) The specified blob does not exist. ContainerAlreadyExists...

Cannot save the file, the remote server returned an error (400 ...

Unable to upload images error is 400: Bad request error when I try to upload an image through the image uploader, or when...