[BUG] BlobBatchClient.submitBatchWithResponse throws BlobStorageException when blobs don't exist and throwOnAnyError flag is false

Describe the bug When submitting a BlobBatch with deleted blobs where all of the blobs are already deleted an BlobStorageException is raised. This happens even though the parameter throwOnAnyFailure is false. According to the api docs a BlobStorageException will only be raised

If the batch request is malformed.

Exception or Stack Trace

com.azure.storage.blob.models.BlobStorageException: <?xml version="1.0" encoding="utf-8"?>
<Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:b9387aaa-301e-0027-075a-fc45b11ed6f6
Time:2020-03-17T12:49:20.2347689Z</Message></Error>

	at com.azure.storage.blob.batch.BlobBatchHelper.setBodyOrAddException(BlobBatchHelper.java:170)
	at com.azure.storage.blob.batch.BlobBatchHelper.lambda$mapBatchResponse$0(BlobBatchHelper.java:107)
	at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:73)
	at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:32)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637)
	at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:419)
	at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:209)
	at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:367)
	at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:363)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:412)
	at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:572)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:90)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:316)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1231)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1268)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:493)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:271)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

To Reproduce

• Create multiple blobs (my tests creates 2)
• Create a blobBatch and add them via deleteBlob(container, blobName)
• Delete them via BlobBatchClient.submitBatchWithResponse(blobBatch, false, null, Context.NONE)

Code Snippet ~Code is in Scala, let me know if I should rewrite to Java.~ Rewritten in Java

package co.famly;

import com.azure.core.http.rest.Response;
import com.azure.core.util.Context;
import com.azure.storage.blob.BlobClient;
import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;
import com.azure.storage.blob.batch.BlobBatch;
import com.azure.storage.blob.batch.BlobBatchClient;
import com.azure.storage.blob.batch.BlobBatchClientBuilder;
import com.azure.storage.blob.models.BlobHttpHeaders;

import java.io.ByteArrayInputStream;
import java.util.HashMap;
import java.util.UUID;

public class Main {

    public static void main(String[] args) {
        String connectionString = "myConnectionString";
        String containerName = "test-container";
        String firstBlob = "test-data/" + UUID.randomUUID();
        String secondBlob ="test-data/" + UUID.randomUUID();

        BlobServiceClient client = new BlobServiceClientBuilder().connectionString(connectionString).buildClient();
        BlobContainerClient container = client.getBlobContainerClient(containerName);
        BlobBatchClient batchClient = new BlobBatchClientBuilder(client).buildClient();

        uploadBlob(container.getBlobClient(firstBlob));
        uploadBlob(container.getBlobClient(secondBlob));

        // Second iteration of loop should result in the unexpected BlobStorageException
        for (int i = 0; i < 2; i++) {
            BlobBatch batch = batchClient.getBlobBatch();
            Response<Void> firstResponse = batch.deleteBlob(containerName, firstBlob);
            Response<Void> secondResponse = batch.deleteBlob(containerName, secondBlob);
            // Batch response is Null for some reason, maybe related to https://github.com/Azure/azure-sdk-for-java/issues/7256
            Response<Void> batchResponse = batchClient.submitBatchWithResponse(batch, false, null, Context.NONE);
            System.out.println(String.format("iteration %d: firstStatus: %d secondStatus: %d", i, firstResponse.getStatusCode(), secondResponse.getStatusCode()));
        }
    }

    private static void uploadBlob(BlobClient blob) {
        BlobHttpHeaders headers = new BlobHttpHeaders().setContentType("text/plain");
        blob.uploadWithResponse(
            new ByteArrayInputStream(testData.getBytes()),
            testData.getBytes().length,
            null,
            headers,
            new HashMap<>(),
            null,
            null,
            null,
            null
        );
    }

    private static String testData = "This is some test data";
}

Running above program with a valid connection string to a Blob Storage account should exhibit the bug.

Expected behavior The call to submitBatchWithResponse to not throw an exception when the individual sub-requests are valid, albeit will fail, and the throwOnAnyFailure flag is set to false. Instead I would expect the result of each sub-request to be reflected in the Response<Void> objects that were returned when the deleteBlob method was invoked on the BlobBatch object.

Setup (please complete the following information):

• OS: macOS 10.15.3
• IDE : IntelliJ
• azure-storage-blob:12.5.0 azure-storage-blob-batch:12.4.0-beta1

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

EDIT: Updated reproduction code to be Java

[rickle-msft]

@kalanzai Thanks for understanding 😃 Likewise, your initial confusion at this behavior is reasonable, and we will keep this in mind for possible future additions where maybe it will become prudent for us to add an option to disable errors even on accessing the sub responses. From my investigation, it seems like this would be fairly straightforward.

I am going to close this issue now as I believe it has been resolved, but please feel free to continue engaging in whatever manner suits you if you need further support.

For future reference, in case this item gets picked up, we could feasibly achieve this behavior by updating the following code in BlobBatchHelper to propagate the throwOnAnyFailure flag to the response and then have the response check the flag before throwing:

BlobBatchOperationResponse<?> batchOperationResponse =
                        getBatchOperation(batchOperationInfo, subResponseSections[0], logger);

                    // The second section will contain status code and header information.
                    batchOperationResponse.setStatusCode(getStatusCode(subResponseSections[1], logger));
                    batchOperationResponse.setHeaders(getHttpHeaders(subResponseSections[1]));

[kalanzai]

@rickle-msft Thank you for your response. Living in a Scala world I’m not used to thinking about the fact that accessing a property can have side effects. I understand your design decision of wanting the experience to mimic individual operations and can work around it.