Blob Storage SDK - Server failed to authenticate request after extended timeout

• Package Name: azure-storage-blob
• Package Version: 12.4.0
• Operating System: Ubuntu 18.04
• Python Version: 3.6.6

Describe the bug Running a read intense application and at some point of time, a time out happens while the application is trying to read from the blob storage (we have an active incident to identify why this is happening).

TimeoutError: [Errno 110] Connection timed 

Timeout lasts for approximately 15minutes and then while the codebase tries to recover a new exception happens from the storage SDK with the following message:

AuthenticationErrorDetail:Request date header too old: DATE OF TIMED OUT REQUEST

To Reproduce Not sure how you could cause the 15minute timeout but perhaps the following can replicate results:

1. Make a request to read a blob through proxy
2. Pause the request in the proxy
3. When the timeout happens, allow the retry request to hit the storage account.

Expected behavior The Request date header too old exceptions should not be thrown.

Stack trace

Fatal read error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f1cf667a5c0>                 
transport: <_SelectorSocketTransport fd=121 read=polling write=<idle, bufsize=0>> 
Traceback (most recent call last):                                                                                                            
  File "/home/azureuser/genfiles/external/python_runtime/python3/lib/python3.6/asyncio/selector_events.py", line 727, in _read_ready
    data = self._sock.recv(self.max_size)
TimeoutError: [Errno 110] Connection timed out                                                                                                                                                                    
ERROR    2020-09-08 16:08:45,961 customclass load_blob_file status=error, duration_ms=958266.8999999999
Traceback (most recent call last):
  File "/home/azureuser/bin/azureuser/azure/storage/blob/aio/_download_async.py", line 271, in _initial_request
    **self._request_options)
  File "/home/azureuser/bin/azureuser/azure/storage/blob/_generated/aio/operations_async/_blob_operations_async.py", line 169, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/azureuser/bin/azureuser/services/storage/blob_storage/blob_loader_storage.py", line 128, in load_blob_file
    file = self._hot_storage.load_file(file_id)
  File "/home/azureuser/bin/azureuser/services/storage/blob_storage/azure_blob_storage.py", line 60, in load_file
    loop=self._get_or_create_event_loop(),
  File "/home/azureuser/genfiles/external/python_runtime/python3/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/home/azureuser/genfiles/external/python_runtime/python3/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/azureuser/bin/azureuser/services/storage/blob_storage/azure_blob_storage.py", line 79, in _load_blob_async_into_byte_stream
    storage_stream_downloader = await blob_client.download_blob()
  File "/home/azureuser/bin/azureuser/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer 
    return await func(*args, **kwargs)
  File "/home/azureuser/bin/azureuser/azure/storage/blob/aio/_blob_client_async.py", line 335, in download_blob
    await downloader._setup()  # pylint: disable=protected-access
  File "/home/azureuser/bin/azureuser/azure/storage/blob/aio/_download_async.py", line 225, in _setup
    self._response = await self._initial_request()                                                    
  File "/home/azureuser/bin/azureuser/azure/storage/blob/aio/_download_async.py", line 306, in _initial_request
    process_storage_error(error)
  File "/home/azureuser/bin/azureuser/azure/storage/blob/_shared/response_handlers.py", line 147, in process_storage_error
    raise error
azure.core.exceptions.ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:4a1422e8-101e-0039-72fa-85b01b000000
Time:2020-09-08T16:08:45.9594457Z
ErrorCode:AuthenticationFailed
Error:None
AuthenticationErrorDetail:Request date header too old: 'Tue, 08 Sep 2020 15:52:47 GMT'
ERROR    2020-09-08 16:08:45,963 customclass loader_fetch_batch_load_data status=error, duration_ms=958596.6
Traceback (most recent call last):
  File "/home/azureuser/bin/azureuser/azure/storage/blob/aio/_download_async.py", line 271, in _initial_request
    **self._request_options)
  File "/home/azureuser/bin/azureuser/azure/storage/blob/_generated/aio/operations_async/_blob_operations_async.py", line 169, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.'

[vbarbaresi]

I went a bit further in the investigation:

Why does it timeout despite setting 1 second timeout in the client

I put a logger in this place: https://github.com/Azure/azure-sdk-for-python/blob/0e82fae6b881d39497f8eb6890a33969dd9eec31/sdk/core/azure-core/azure/core/pipeline/transport/_requests_basic.py#L258 to get the actual timeout that was used.

I tried with my code (on azure queue), and with your (blob storage). We both use DefaultAzureCredential

The same thing happens: 3 requests are sent in total, the first 2 requests to get a token (from http://169.254.169.254/metadata/identity/oauth2/token)

timeout (0.3, 300) to http://169.254.169.254/metadata/identity/oauth2/token`
timeout (604800, 300) to http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com
timeout (1,1) to "https://<service>.core.windows.net"

So the first 2 requests have a timeout of 300 seconds, and I’m not sure how to override that. The tokens are valid for an hour, so the client only sends these requests once an hour, but if it fails the timeout is very long.

Why does it take more than 15 min to timeout:

It’s because of the default retry policy which is ExponentialRetry.

https://github.com/Azure/azure-sdk-for-python/blob/0f9c3be0bf4557daea0c83e286b7fc051c20625d/sdk/storage/azure-storage-blob/azure/storage/blob/_shared/policies_async.py#L138-L142

So it retries 3 times, the backoff delay is (in average, there is some jitter): 15 + 18 + 24 = 57 seconds

Added to the 3 times * 300 seconds timeout, it makes 15 min 57 seconds. Which is almost exactly the duration in the initial report:

AuthenticationErrorDetail:Request date header too old: 'Tue, 08 Sep 2020 15:52:47 GMT'
ERROR    2020-09-08 16:08:45,963 customclass loader_fetch_batch_load_data status=error, duration_ms=958596.6

(15 min 58 seconds) And I observed this duration multiple times as well.

Summary

• We should be able to configure the timeout of the IMDS queries: 300 seconds default is way too long.
• The request Date header should be updated during the retries, because eventually the retry fails because the header is too old

[vbarbaresi]

interesting… nice work on the reproduction script. Maybe it’s doing a backoff retry, and during these retries it doesn’t update the request date? I’ll wait for Azure support to answer.

I should add that in our case, there is no reason for 15 minute timeouts. It looks a bit like we’re being throttled