question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] NullReferenceException thrown from Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync when specifying high StorageTransferOptions.MaximumConcurrency

See original GitHub issue

Describe the bug This appears like a regression in the latest version (does not reproduce in 12.2.2):

System.NullReferenceException: Object reference not set to an instance of an object.
   at Azure.Storage.Files.DataLake.DataLakeFileClient.<>c__DisplayClass83_0.<<GetPartitionedUploaderBehaviors>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Storage.PartitionedUploader`2.StagePartitionAndDisposeInternal(SlicedStream partition, Int64 offset, TServiceSpecificArgs args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.PartitionedUploader`2.UploadInParallelAsync(Stream content, Nullable`1 contentLength, Int64 blockSize, TServiceSpecificArgs args, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.PartitionedUploader`2.UploadInParallelAsync(Stream content, Nullable`1 contentLength, Int64 blockSize, TServiceSpecificArgs args, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.PartitionedUploader`2.UploadInternal(Stream content, TServiceSpecificArgs args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.StagedUploadInternal(Stream content, DataLakeFileUploadOptions options, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(String path, DataLakeFileUploadOptions options, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(String path, PathHttpHeaders httpHeaders, DataLakeRequestConditions conditions, IProgress`1 progressHandler, StorageTransferOptions transferOptions, CancellationToken cancellationToken)

To Reproduce Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

await fileClient.UploadAsync(someLargeFileLike2GB, transferOptions: new StorageTransferOptions
            { 
                MaximumConcurrency = 500,
                InitialTransferLength = 1024 * 1024 * 4,
                MaximumTransferLength = 1024 * 1024 * 4
            })

Environment:

  • Name and version of the Library package used: Azure.Storage.Files.DataLake 12.3.1
  • Hosting platform or OS and .NET runtime version:
.NET Core SDK (reflecting any global.json):
 Version:   3.1.401
 Commit:    5b6f5e5005

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.401\

Host (useful for support):
  Version: 3.1.7
  Commit:  fcfdef8d6b

.NET Core SDKs installed:
  1.1.11 [C:\Program Files\dotnet\sdk]
  1.1.14 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.503 [C:\Program Files\dotnet\sdk]
  2.1.514 [C:\Program Files\dotnet\sdk]
  2.1.515 [C:\Program Files\dotnet\sdk]
  3.1.302 [C:\Program Files\dotnet\sdk]
  3.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  • IDE and version : Visual Studio 16.7.2

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:7 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
sumantmehtamscommented, Sep 2, 2020

Adding @seanmcc-msft to take a look.

1reaction
jsquirecommented, Aug 21, 2020

Thank you for your feedback. Tagging and routing to the team best able to assist.

Read more comments on GitHub >

github_iconTop Results From Across the Web

DataLakeFileClient.UploadAsync Method - Storage
The UploadAsync(Stream, Boolean, CancellationToken) operation creates and uploads content to a file. If the overwrite parameter is not specified and the file ......
Read more >
DataLakeFileClient Class (Azure.Storage.Files.DataLake)
The DataLakeFileClient allows you to manipulate Azure Data Lake files. ... To upload large files all at once, consider using UploadAsync(Stream).
Read more >
Tuning your uploads and downloads with the Azure ...
Here, you define values for the following properties, which are the basis for managing your transfer: MaximumConcurrency : the maximum number ...
Read more >
BlobClient.UploadAsync Method (Azure.Storage.Blobs)
The UploadAsync(BinaryData, Boolean, CancellationToken) operation creates a new block blob or throws if the blob already exists. Setting overwrite to true ...
Read more >
DataLakeFileClient.MaxUploadLongBytes Property
Gets the maximum number of bytes that can be sent in each append call in Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(System.IO.Stream,Azure.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found