[BUG] Azure storage batch API does not include all blobs in the request

Describe the bug Sometimes the batch API SDK does not include all the blobs specified with the batch in the request body. Also, it throws a different exception type (BlobBatchStorageException instead of BlobStorageException) when deletion for one of the included blobs fails.

Exception or Stack Trace Here are the logs from the SDK. As you can see, there are two delete requests logged when 2 blobs were added to the batch. However, when the batch is submitted, there is only one blob in the request body. And when that one blob was not found, it threw BlobStorageException instead of BlobBatchStorageException.

[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - --> DELETE https://<account>.blob.core.windows.net/sdffd/FDN%20Storage%20Accounts.txt
[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - Date:Mon, 09 Dec 2019 22:29:14 GMT
Authorization:REDACTED
Content-Length:0
x-ms-delete-snapshots:include
Content-Id:REDACTED
x-ms-client-request-id:6387d5cd-328f-4c2c-9a31-442ce54a1a50
User-Agent:azsdk-java-azure-storage-blob/12.0.0 12.0.2; Linux 3.10.0-1062.4.1.el7.x86_64

[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - (empty body)
[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - --> END DELETE
[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - --> DELETE https://<account>.blob.core.windows.net/sdffd/vcbcvsfsfdfipui
[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - Date:Mon, 09 Dec 2019 22:29:14 GMT
Authorization:REDACTED
Content-Length:0
x-ms-delete-snapshots:include
Content-Id:REDACTED
x-ms-client-request-id:415bfe91-b368-4d50-9876-66d1a9e0645a
User-Agent:azsdk-java-azure-storage-blob/12.0.0 12.0.2; Linux 3.10.0-1062.4.1.el7.x86_64

[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - (empty body)
[main] INFO com.azure.storage.blob.implementation.BlobsImpl$BlobsService.delete - --> END DELETE
[main] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - --> POST https://<account>.blob.core.windows.net?comp=batch
[main] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - comp=batch
[main] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - Date:Mon, 09 Dec 2019 22:29:14 GMT
Authorization:REDACTED
Content-Length:559
x-ms-version:2019-02-02
Content-Type:multipart/mixed; boundary=batch_026fbdee-89e2-427f-a3d0-2827bf9fa2c9
x-ms-client-request-id:a71da994-301f-43ea-ad24-f10cb7507662
User-Agent:azsdk-java-azure-storage-blob/12.0.0 12.0.2; Linux 3.10.0-1062.4.1.el7.x86_64

[parallel-5] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - 559-byte body:%n--batch_026fbdee-89e2-427f-a3d0-2827bf9fa2c9
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: 0

DELETE /sdffd/FDN%20Storage%20Accounts.txt HTTP/1.1
Date: Mon, 09 Dec 2019 22:29:14 GMT
Authorization: SharedKey <account>:<key>
Content-Length: 0
x-ms-delete-snapshots: include
x-ms-client-request-id: 6387d5cd-328f-4c2c-9a31-442ce54a1a50
User-Agent: azsdk-java-azure-storage-blob/12.0.0 12.0.2; Linux 3.10.0-1062.4.1.el7.x86_64

--batch_026fbdee-89e2-427f-a3d0-2827bf9fa2c9--

[parallel-5] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - --> END POST
[reactor-http-epoll-1] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - <-- 202 https://<account>.blob.core.windows.net?comp=batch (643 ms, unknown-length body)
[reactor-http-epoll-1] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - Date:Mon, 09 Dec 2019 22:29:14 GMT
Server:Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
Transfer-Encoding:chunked
x-ms-version:2019-02-02
Content-Type:multipart/mixed; boundary=batchresponse_dc91c5ff-b7e8-4d54-b82f-b25314e64579
x-ms-request-id:2ed98cea-901e-0053-74e0-ae9a38000000
x-ms-client-request-id:a71da994-301f-43ea-ad24-f10cb7507662

[reactor-http-epoll-1] WARN com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - Could not parse the HTTP header content-length: 'null'.
[reactor-http-epoll-1] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - (body content not logged)
[reactor-http-epoll-1] INFO com.azure.storage.blob.implementation.ServicesImpl$ServicesService.submitBatch - <-- END HTTP
[reactor-http-epoll-1] ERROR com.azure.storage.blob.batch.BlobBatchAsyncClient - BlobNotFound
Exception in thread "main" com.azure.storage.blob.models.BlobStorageException: BlobNotFound
	at com.azure.storage.blob.batch.BlobBatchHelper.lambda$null$0(BlobBatchHelper.java:83)
	at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:73)
	at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:32)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1592)
	at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:397)
	at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:197)
	at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:345)
	at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:363)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:412)
	at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:556)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:835)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:93)
		at reactor.core.publisher.Mono.block(Mono.java:1663)
		at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:94)
		at com.azure.storage.blob.batch.BlobBatchClient.submitBatchWithResponse(BlobBatchClient.java:87)
		at com.azure.storage.blob.batch.BlobBatchClient.submitBatch(BlobBatchClient.java:60)
                 ...<application stack>...

To Reproduce I don’t have a very consistent way to reproduce it. I’m able to repro it some times with the following sequence:

• Add one existing blob and non-existing blob to the list of blobs to be deleted in the below snippet.
• When you execute it the first time, one blob will get deleted and the other one will fail and you should see a BlobBatchStorageException.
• Now both deletes should fail in the 2nd attempt with a BlobBatchStorageException.
• Execute the same snippet a couple more times and you might see the behavior I described.

Code Snippet

String containerName = "<container>";

List<String> blobsToDelete = new ArrayList<>();
blobsToDelete.add("<existing blob>");
blobsToDelete.add("<non-existing blob>");

BlobBatchClient batchClient = new BlobBatchClientBuilder(blobServiceClient).buildClient();

BlobBatch batchRequest = batchClient.getBlobBatch();

for (String blob : blobsToDelete)
{
        batchRequest.deleteBlob(
                containerName,
                blob,
                DeleteSnapshotsOptionType.INCLUDE,
                null);
}

batchClient.submitBatch(batchRequest);

Expected behavior

• The SDK should include all the blobs specified in the batch when sending the request to the server.
• The SDK should always throw BlobBatchStorageException when one or more requests in the batch fail.

Screenshots NA

Setup (please complete the following information):

• OS: Linux
• IDE: IntelliJ
• Version of the Library used: 12.0.0

Additional context NA

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[alzimmermsft]

This issue should be resolved by the most recently release version of Azure Storage Blob Batch.

[alzimmermsft]

Resolved by PR #6882