[BUG] Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync with low upload speed - RequestFailedException: OperationTimedOut (500) - Operation could not be completed within the specified time

Azure.RequestFailedException: Operation could not be completed within the specified time.
RequestId:7ea85520-601f-0095-4c19-79a760000000
Time:2020-08-23T06:51:33.3743708Z
Status: 500 (Operation could not be completed within the specified time.)
ErrorCode: OperationTimedOut

Headers:
Server: Windows-Azure-HDFS/1.0,Microsoft-HTTPAPI/2.0
x-ms-error-code: OperationTimedOut
x-ms-request-id: 7ea85520-601f-0095-4c19-79a760000000
x-ms-version: 2019-07-07
x-ms-client-request-id: b5ab9d35-f982-4316-9da1-3046c8882d0d
Date: Sun, 23 Aug 2020 06:51:32 GMT
Content-Length: 193
Content-Type: application/json; charset=utf-8

   at Azure.Storage.Files.DataLake.DataLakeRestClient.Path.AppendDataAsync_CreateResponse(ClientDiagnostics clientDiagnostics, Response response)
   at Azure.Storage.Files.DataLake.DataLakeRestClient.Path.AppendDataAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, Stream body, String version, Nullable`1 position, Nullable`1 timeout, Nullable`1 contentLength, Byte[] transactionalContentHash, String leaseId, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.AppendInternal(Stream content, Nullable`1 offset, Byte[] contentHash, String leaseId, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.AppendAsync(Stream content, Int64 offset, Byte[] contentHash, String leaseId, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakePartitionedUploader.AppendBlockAsync(ChunkedStream block, Int64 offset, String leaseId, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakePartitionedUploader.UploadInParallelAsync(Stream content, Int32 blockSize, PathHttpHeaders httpHeaders, DataLakeRequestConditions conditions, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakePartitionedUploader.UploadInParallelAsync(Stream content, Int32 blockSize, PathHttpHeaders httpHeaders, DataLakeRequestConditions conditions, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakePartitionedUploader.UploadAsync(Stream content, PathHttpHeaders httpHeaders, IDictionary`2 metadata, String permissions, String umask, DataLakeRequestConditions conditions, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.StagedUploadAsync(Stream content, PathHttpHeaders httpHeaders, IDictionary`2 metadata, String permissions, String umask, DataLakeRequestConditions conditions, IProgress`1 progressHandler, StorageTransferOptions transferOptions, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(String path, PathHttpHeaders httpHeaders, DataLakeRequestConditions conditions, IProgress`1 progressHandler, StorageTransferOptions transferOptions, CancellationToken cancellationToken)

To Reproduce Call Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync with a ~300MB file using a slow connection (< 2Mbps upload - note mega bits per seconds)

Environment:

• Name and version of the Library package used: Azure.Storage.Files.DataLake 12.2.2
• Hosting platform or OS and .NET runtime version:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.401
 Commit:    5b6f5e5005

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.401\

Host (useful for support):
  Version: 3.1.7
  Commit:  fcfdef8d6b

.NET Core SDKs installed:
  1.1.11 [C:\Program Files\dotnet\sdk]
  1.1.14 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.503 [C:\Program Files\dotnet\sdk]
  2.1.514 [C:\Program Files\dotnet\sdk]
  2.1.515 [C:\Program Files\dotnet\sdk]
  3.1.302 [C:\Program Files\dotnet\sdk]
  3.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• IDE and version : Visual Studio 16.7.2

[sumantmehtams]

Adding @seanmcc-msft to take a look.

[jsquire]

Thank you for your feedback. Tagging and routing to the team best able to assist.