Asynchronously processing GetObjectAsync stream

Feature request:

GetObjectAsync takes a synchronous stream processor. It defeats the point of the call being asynchronous, unless I’m misunderstanding something (is the stream buffered in memory already?)

Task GetObjectAsync(string bucketName, string objectName, Action<Stream> callback, ServerSideEncryption sse = null, CancellationToken cancellationToken = default(CancellationToken))

It would be great to see an overload like:

Task GetObjectAsync(string bucketName, string objectName, Func<Stream, Task> callback, ServerSideEncryption sse = null, CancellationToken cancellationToken = default(CancellationToken))

Notice how callback is changed Action<Stream> callback -> Func<Stream, Task> callback.

For the time being, I came up with a workaround using TaskCompletionSource, although I cannot test if it is truly asynchronous or blocking.

                var completion = new TaskCompletionSource<object>();

                await _client.GetObjectAsync(_options.BucketName, name, async s =>
                {
                    await s.CopyToAsync(file.Stream, cancellationToken);

                    completion.TrySetResult(null);
                }, null, cancellationToken);

                await completion.Task;

[Biztactix-Ryan]

@harshavardhana @edemartel @cosminvlad There’s more than one of us sitting here wondering if this is going to get sorted.

Many users will be using this in asp.net of some type, I can only expect that nearly all of them would want to pass data from Minio directly to the user, It would be very rare that we’d need to cache such data. so loading it into a memory stream then writing it to the response stream is double handling and depending on the size of the return may even overrun server memory.

Now if you have already covered this scenario and have an answer, it’d be great to know as I’ve just checked the docs again and it’s still using the callback. If we can’t get such a simple thing sorted, We’ll just move on and it’ll be with another platform.

[cristipufu]

Same here:

Unsuccessful response from server without XML error: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.: Minio.Exceptions.InternalClientException: Minio API responded with message=Unsuccessful response from server without XML error: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
   at void Minio.MinioClient.ParseError(IRestResponse response) in /q/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 450
   at void Minio.MinioClient.HandleIfErrorResponse(IRestResponse response, IEnumerable<ApiResponseErrorHandlingDelegate> handlers, DateTime startTime) in /q/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 505
   at async Task<IRestResponse> Minio.MinioClient.ExecuteTaskAsync(IEnumerable<ApiResponseErrorHandlingDelegate> errorHandlers, IRestRequest request, CancellationToken cancellationToken) in /q/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 352
   at async Task Minio.MinioClient.GetObjectAsync(string bucketName, string objectName, Action<Stream> cb, CancellationToken cancellationToken) in /q/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 56
   at async Task UiPath.Storage.Minio.MinioStorageClient.ReadObjectAsync(StorageLocation location, Stream stream, CancellationToken cancellationToken)

As a workaround we have to enable AllowSynchronousIO in our web app:

services.Configure<IISServerOptions>(options =>
{
    options.AllowSynchronousIO = true;
});

This limitation comes from the underlying RestSharp client itself, I think it’s time to get rid of it and use Microsoft’s HttpClient