[BUG] Batch blob deletion fails with HTTP code 412 when ETag checking is enabled and storage has blob versioning enabled

Posting on behalf of @dacspiir and @adzspiir. We have been asked to open the issue here as the follow-up to the support ticket: TrackingID#2107270050001045

Technical area and API documentation in question Azure SDK and Azure Blob Storage BlobBatch.DeleteBlob

Describe the bug When working with blob storage, submitting a batch deletion where ETag checking is enabled (see example) batch.DeleteBlob("test", item.Name, DeleteSnapshotsOption.None, new BlobRequestConditions { IfMatch = item.Properties.ETag }); will result in response code 412 (The condition specified using HTTP conditional header(s) is not met.)

This only happens when the storage account has the “Enable versioning for blobs” option enabled. Using other values of DeleteSnapshotsOption does not resolve the issue.

Expected behavior Blobs are deleted.

Actual behavior An exception is thrown upon submitting the batch.

Example Response

Headers:
x-ms-error-code: ConditionNotMet
x-ms-request-id: 8def3893-a01e-00a4-4d38-8949fb1e27d5
x-ms-version: 2020-08-04
x-ms-client-request-id: 9a4a0c80-666f-4025-9826-8194286c46da
Content-Length: 253
Content-Type: application/xml
Server: Windows-Azure-Blob/1.0
) (The condition specified using HTTP conditional header(s) is not met.
RequestId:8def3893-a01e-00a4-4d38-8949fb1e27d6
Time:2021-08-04T13:57:13.1333214Z
Status: 412 (The condition specified using HTTP conditional header(s) is not met.)
ErrorCode: ConditionNotMet
 
Content:
<?xml version="1.0" encoding="utf-8"?>
<Error><Code>ConditionNotMet</Code><Message>The condition specified using HTTP conditional header(s) is not met.
RequestId:8def3893-a01e-00a4-4d38-8949fb1e27d6
Time:2021-08-04T13:57:13.1333214Z</Message></Error>

Example Stack Trace

   at Azure.Storage.Blobs.Specialized.BlobBatch.<DeleteBlob>b__23_0(Response response)
   at Azure.Storage.Blobs.Specialized.DelayedResponse.SetLiveResponse(Response live, Boolean throwOnFailure)
   at Azure.Storage.Blobs.Specialized.BlobBatchClient.UpdateOperationResponses(IList`1 messages, Response rawResponse, Stream responseContent, String responseContentType, Boolean throwOnAnyFailure, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Azure.Storage.Blobs.Specialized.BlobBatchClient.UpdateOperationResponses(IList`1 messages, Response rawResponse, Stream responseContent, String responseContentType, Boolean throwOnAnyFailure, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBatchClient.SubmitBatchInternal(BlobBatch batch, Boolean throwOnAnyFailure, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBatchClient.SubmitBatchAsync(BlobBatch batch, Boolean throwOnAnyFailure, CancellationToken cancellationToken)
   at TestBatchDelete.TestClass.DeleteAllBlobs(Boolean triggerBug) in C:\Users\Dani\RiderProjects\TMP\TestBatchDelete\TestBatchDelete\TestClass.cs:line 65
   at TestBatchDelete.TestClass.DeleteAllBlobs(Boolean triggerBug) in C:\Users\Dani\RiderProjects\TMP\TestBatchDelete\TestBatchDelete\TestClass.cs:line 50
   at TestBatchDelete.TestClass.RunTest() in C:\Users\Dani\RiderProjects\TMP\TestBatchDelete\TestBatchDelete\TestClass.cs:line 38
   at TestBatchDelete.Program.Main(String[] args) in C:\Users\Dani\RiderProjects\TMP\TestBatchDelete\TestBatchDelete\Program.cs:line 11
   at TestBatchDelete.Program.<Main>(String[] args)
---> (Inner Exception #1) Azure.RequestFailedException: The condition specified using HTTP conditional header(s) is not met.

To Reproduce The below code is the minimal reproduction case that can be used to trigger the issue assuming the requirements have been met.

csproj

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net5.0</TargetFramework>
    </PropertyGroup>
    <ItemGroup>
      <PackageReference Include="Azure.Storage.Blobs" Version="12.9.1" />
      <PackageReference Include="Azure.Storage.Blobs.Batch" Version="12.6.0" />
    </ItemGroup>
</Project>

.cs

using System;
using System.IO;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Azure.Storage.Blobs.Specialized;

namespace TestBatchDelete
{
    internal class Program
    {
        private static async Task Main(string[] args)
        {
            var tc = new TestClass();
            await tc.RunTest();
        }
    }

    public class TestClass
    {
        private readonly BlobServiceClient _client;

        public TestClass()
        {
            _client = new BlobServiceClient(
                "connectionStringToAStorageAccountWithBlobVersioningEnabled");
        }

        public async Task RunTest()
        {
            // clear out old run (this works)
            await DeleteAllBlobs(false);

            // create demo files
            Console.WriteLine("Creating");
            for (var i = 0; i < 2; i++)
            {
                var memoryStream = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 });
                var returnedInfo = await _client.GetBlobContainerClient("test").GetBlobClient("test" + i)
                    .UploadAsync(memoryStream);
                await memoryStream.DisposeAsync();
            }

            // delete files triggering the bug
            await DeleteAllBlobs(true);
        }

        private async Task DeleteAllBlobs(bool triggerBug)
        {
            Console.WriteLine("Deleting");
            var blobEnumerable = _client
                .GetBlobContainerClient("test")
                .GetBlobsAsync()
                .AsPages(null, 200);

            var batchClient = _client.GetBlobBatchClient();
            await foreach (var t in blobEnumerable)
            {
                if (t.Values.Count == 0)
                    break;
                var batch = batchClient.CreateBatch();
                foreach (var item in t.Values)
                    batch.DeleteBlob("test",
                        item.Name,
                        DeleteSnapshotsOption.None,
                        // BUG: Here below if we specify a matching etag (triggerBug=true) then the blobs will not be deleted.
                        new BlobRequestConditions { IfMatch = triggerBug ? item.Properties.ETag : null });

                // here blob storage service returns a "precondition failed" error instead of deleting the blobs.
                await batchClient.SubmitBatchAsync(batch, true);
            }
        }
    }
}

Environment:

• Azure.Storage.Blobs, 12.9.1
• Azure.Storage.Blobs.Batch, 12.6.0
• dotnet, 5.0.201
• Rider 2021.2, macOS Big Sur

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 92a1ce81-8222-ede3-2afb-63fc6ac19fee
• Version Independent ID: 9af6b596-9e70-d49a-1f6d-2b87bbc89128
• Content: BlobBatch.DeleteBlob Method (Azure.Storage.Blobs.Specialized) - Azure for .NET Developers
• Content Source: xml/Azure.Storage.Blobs.Specialized/BlobBatch.xml
• Service: azure
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

[hiteshsharma-msft]

@dacspiir Storage team is actively working on it. We will update you asap. As a workaround, Will it be feasible, in your scenario, to directly delete the blob (https://docs.microsoft.com/en-us/dotnet/api/azure.storage.blobs.blobcontainerclient.deleteblob?view=azure-dotnet) instead of blobbatchclient?

[dacspiir]

Hello, After spending quite a bit of time in investigating this issue and setting up your test framework to use live resources, I finally managed to get to what seemed to be the problem that made the test case difficult to reproduce. It seems that the first ever batch delete operation will work, but all the successive ones on the same container will all fail.

Here is a test for you that reproduces the issue in your test suite.

        [RecordedTest]
        public async Task DAC_Test3()
        {
            BlobServiceClient blobServiceClient = GetServiceClient_SharedKey();
            await using DisposingContainer test = await GetTestContainerAsync();

            async Task RunTestOnce()
            {
                AppendBlobClient appendBlobClient = InstrumentClient(test.Container.GetAppendBlobClient("hey"));
                await appendBlobClient.CreateAsync();

                List<BlobItem> blobItems = new List<BlobItem>();
                await foreach (BlobItem blobItem in test.Container.GetBlobsAsync())
                {
                    blobItems.Add(blobItem);
                }

                BlobBatchClient blobBatchClient = blobServiceClient.GetBlobBatchClient();
                using BlobBatch batch = blobBatchClient.CreateBatch();
                Response[] responses = new Response[]
                {
                    batch.DeleteBlob(test.Container.Name, blobItems[0].Name, DeleteSnapshotsOption.None,
                        new BlobRequestConditions {IfMatch = blobItems[0].Properties.ETag})
                };
                Response response = await blobBatchClient.SubmitBatchAsync(batch, true);
            }

            await RunTestOnce(); // this works
            await RunTestOnce(); // this doesn't, as any new batch delete operation will fail after the first one.
        }

The issue was hidden in the test suite because the containers are not reused and so it was not triggering the bug. Our test code was instead triggering the bug.

This now seems very much a backend issue, so this bug might be the not appropriate place to report this, if this is the case can you please direct me or forward this to the appropriate team? Thank you!