[Storage] Retries exceed MaxRetries if server sends partial response

When using Azure.Core@1.1.0 and Azure.Storage.Blobs@12.4.0-dev.20200306.1, BlobClient.DownloadToAsync() appears to retry indefinitely if the server sends a partial body, even though MaxRetries = 1.

Repro Steps

1. Start a server which can inject HTTP faults, like https://github.com/mikeharder/HttpFaultInjector.

2. Create a BlobClient with the following options:

var httpClient = new HttpClient(httpClientHandler)
{
    Timeout = TimeSpan.FromSeconds(5)
};

var blobClientOptions = new BlobClientOptions()
{
    Transport = new ChangeUriTransport(new HttpClientTransport(httpClient), "localhost", 7778),
    Retry = {
        MaxRetries = 1,
        NetworkTimeout = TimeSpan.FromSeconds(10)
    },
};

var blobClient = new BlobClient(connectionString, "testcontainer", "testblob", blobClientOptions);
var response = await blobClient.DownloadToAsync(new MemoryStream());

private class ChangeUriTransport : HttpPipelineTransport
{
    private readonly HttpPipelineTransport _transport;
    private readonly string _host;
    private readonly int? _port;

    public ChangeUriTransport(HttpPipelineTransport transport, string host, int? port)
    {
        _transport = transport;
        _host = host;
        _port = port;
    }

    public override Request CreateRequest()
    {
        return _transport.CreateRequest();
    }

    public override void Process(HttpMessage message)
    {
        ChangeUri(message);
        _transport.Process(message);
    }

    public override ValueTask ProcessAsync(HttpMessage message)
    {
        ChangeUri(message);
        return _transport.ProcessAsync(message);
    }

    private void ChangeUri(HttpMessage message)
    {
        // Ensure Host header is only set once, since the same HttpMessage will be reused on retries
        if (!message.Request.Headers.Contains("Host"))
        {
            message.Request.Headers.Add("Host", message.Request.Uri.Host);
        }

        message.Request.Uri.Host = _host;
        if (_port.HasValue)
        {
            message.Request.Uri.Port = _port.Value;
        }
    }
}

1. Run the client app, and instruct the server to send a partial response (say 50% of the body).

2. The client will read the partial response, block for 10 seconds (NetworkTimeout), then retry but only requesting the remaining 50% of bytes (with a Content-Range header).

3. Again instruct the server to send a partial response (50% of the remaining body == 25% of the original body).

4. The client will again read the partial response, block for 10 seconds (NetworkTimeout), then retry but only requesting the remaining 25% of original bytes.

5. This can be repeated indefinitely, despite MaxRetries=1. I’m not sure if this is a bug or by design.

[tg-msft]

So my understanding of this issue is that we’ll do the following:

1. Start downloading with the customer’s MaxRetries via the pipeline
2. Wrap in a RetriableStream
3. If downloading the stream fails after we’ve left the pipeline a. Try to start downloading from where we left off with the customer’s MaxRetries via the pipeline b. If that succeeds, go back to step 2 c. If that fails, loop back to step 3, up to 3 times
4. Return the downloaded blob

We don’t adjust MaxRetries in Step 3a to account for the number of attempts in Steps 1 or previous iterations of Step 3a. We also don’t keep a running number of attempts in Step 3c so every stream failure kicks off a whole new cycle. This could take a very long time under adverse circumstances (absolute worst case being 3 * MaxRetries * BlobSize requests if Mike sets up a test server that fails after sending back a single byte).

Mike’s point in raising the issue is number of actual retries that could happen in a flaky blob download doesn’t correspond to how the customer envisions MaxRetries working. Pavel and I are okay with this until we hear from customers at which point we could add a few more RetryPolicy knobs to allow configuring what happens in Step 3.

[pakrym]

Customers never reported problems with this behavior.