[BUG] NullReferenceException thrown from Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync when specifying high StorageTransferOptions.MaximumConcurrency
See original GitHub issueDescribe the bug This appears like a regression in the latest version (does not reproduce in 12.2.2):
System.NullReferenceException: Object reference not set to an instance of an object.
at Azure.Storage.Files.DataLake.DataLakeFileClient.<>c__DisplayClass83_0.<<GetPartitionedUploaderBehaviors>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Azure.Storage.PartitionedUploader`2.StagePartitionAndDisposeInternal(SlicedStream partition, Int64 offset, TServiceSpecificArgs args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.PartitionedUploader`2.UploadInParallelAsync(Stream content, Nullable`1 contentLength, Int64 blockSize, TServiceSpecificArgs args, IProgress`1 progressHandler, CancellationToken cancellationToken)
at Azure.Storage.PartitionedUploader`2.UploadInParallelAsync(Stream content, Nullable`1 contentLength, Int64 blockSize, TServiceSpecificArgs args, IProgress`1 progressHandler, CancellationToken cancellationToken)
at Azure.Storage.PartitionedUploader`2.UploadInternal(Stream content, TServiceSpecificArgs args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Files.DataLake.DataLakeFileClient.StagedUploadInternal(Stream content, DataLakeFileUploadOptions options, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(String path, DataLakeFileUploadOptions options, CancellationToken cancellationToken)
at Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(String path, PathHttpHeaders httpHeaders, DataLakeRequestConditions conditions, IProgress`1 progressHandler, StorageTransferOptions transferOptions, CancellationToken cancellationToken)
To Reproduce Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)
await fileClient.UploadAsync(someLargeFileLike2GB, transferOptions: new StorageTransferOptions
{
MaximumConcurrency = 500,
InitialTransferLength = 1024 * 1024 * 4,
MaximumTransferLength = 1024 * 1024 * 4
})
Environment:
- Name and version of the Library package used:
Azure.Storage.Files.DataLake 12.3.1
- Hosting platform or OS and .NET runtime version:
.NET Core SDK (reflecting any global.json):
Version: 3.1.401
Commit: 5b6f5e5005
Runtime Environment:
OS Name: Windows
OS Version: 10.0.18363
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.401\
Host (useful for support):
Version: 3.1.7
Commit: fcfdef8d6b
.NET Core SDKs installed:
1.1.11 [C:\Program Files\dotnet\sdk]
1.1.14 [C:\Program Files\dotnet\sdk]
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.503 [C:\Program Files\dotnet\sdk]
2.1.514 [C:\Program Files\dotnet\sdk]
2.1.515 [C:\Program Files\dotnet\sdk]
3.1.302 [C:\Program Files\dotnet\sdk]
3.1.401 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
- IDE and version : Visual Studio 16.7.2
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:7 (2 by maintainers)
Top Results From Across the Web
DataLakeFileClient.UploadAsync Method - Storage
The UploadAsync(Stream, Boolean, CancellationToken) operation creates and uploads content to a file. If the overwrite parameter is not specified and the file ......
Read more >DataLakeFileClient Class (Azure.Storage.Files.DataLake)
The DataLakeFileClient allows you to manipulate Azure Data Lake files. ... To upload large files all at once, consider using UploadAsync(Stream).
Read more >Tuning your uploads and downloads with the Azure ...
Here, you define values for the following properties, which are the basis for managing your transfer: MaximumConcurrency : the maximum number ...
Read more >BlobClient.UploadAsync Method (Azure.Storage.Blobs)
The UploadAsync(BinaryData, Boolean, CancellationToken) operation creates a new block blob or throws if the blob already exists. Setting overwrite to true ...
Read more >DataLakeFileClient.MaxUploadLongBytes Property
Gets the maximum number of bytes that can be sent in each append call in Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(System.IO.Stream,Azure.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Adding @seanmcc-msft to take a look.
Thank you for your feedback. Tagging and routing to the team best able to assist.