[BUG] Azure Data Lake SDK throws a JsonReaderException if a non-successful “ListPaths” response contains a malformed JSON body

Library name and version

Azure.Storage.Files.DataLake 12.12.1 (& earlier)

Describe the bug

The SDK does not seem to gracefully handle responses that contain a malformed JSON body. We’ve encountered this situation in our production environment.

Consider the following code sample*, in which I simulate a response with malformed body by using DataLakeClientOptions.Transport:

static int GetNumberOfPathsTest(Uri adlDirecetoryUri, TokenCredential tokenCredential)
{
    var handler = new SimulateResponseWithMalformedJsonHttpClientHandler();

    var client = new DataLakeDirectoryClient(adlDirecetoryUri, tokenCredential,
        new DataLakeClientOptions
        {
            Transport = new HttpClientTransport(handler)
        });

    return client.GetPaths().Count();
}

public class SimulateResponseWithMalformedJsonHttpClientHandler : HttpClientHandler
{
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var response = new HttpResponseMessage(HttpStatusCode.InternalServerError);
        response.Content = new StringContent("{Malformed JSON}", Encoding.UTF8, "application/json");
        return Task.FromResult(response);
    }
}

When executed, the SDK throws the following exception:

System.Text.Json.JsonReaderException: 'M' is an invalid start of a property name. Expected a '"'. LineNumber: 0 | BytePositionInLine: 1.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, Utf8JsonReader reader, MetadataDb& database, StackRowStack& stack)
   at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedBytes)
   at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 json, JsonDocumentOptions options)
   at System.Text.Json.JsonDocument.Parse(String json, JsonDocumentOptions options)
   at Azure.Core.Pipeline.StorageClientDiagnostics.ExtractFailureContent(String content, ResponseHeaders responseHeaders, IDictionary`2& additionalInfo)
   at Azure.Core.Pipeline.ClientDiagnostics.CreateRequestFailedException(Response response, ResponseError error, IDictionary`2 additionalInfo, Exception innerException)
   at Azure.Storage.Files.DataLake.FileSystemRestClient.ListPaths(Boolean recursive, Nullable`1 timeout, String continuation, String path, Nullable`1 maxResults, Nullable`1 upn, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileSystemClient.<GetPathsInternal>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Files.DataLake.Models.GetPathsAsyncCollection.<GetNextPageAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
   at Azure.Storage.StorageCollectionEnumerator`1.StoragePageable.<GetEnumerator>d__5.MoveNext()
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)
   at ConsoleApp1.Program.GetNumberOfPathsTest(Uri adlDirecetoryUri, TokenCredential tokenCredential)

Looking at the source code, it seems that StorageClientDiagnostics is overriding its base class’s ExtractFailureContent method, but unlike the overridden method, it is not wrapped with a try-catch block. This means that if a non-successful response contains a malformed JSON body, the call to JsonDocument.Parse(content) will throw an exception which will not be handled.

*Note that the code sample above won’t work if compiled in .NET Core, due to a different unrelated issue: #34466

Expected behavior

Malformed server responses should be handled gracefully.

Actual behavior

The SDK throws a System.Text.Json.JsonReaderException (see details above).

Reproduction Steps

Compile the code snippet above using .NET Framework and execute. Note that the code sample above won’t work if compiled in .NET Core, due to a different unrelated issue: #34466

Environment

No response

[JoshLove-msft]

One thing to note is that the referenced type StorageClientDiagnostics doesn’t exist anymore - it was deleted in https://github.com/Azure/azure-sdk-for-net/pull/33001. The issue is that the old diagnostics code did not have robust handling for when the response from the service was not valid based on the content type. The new code does a try/catch so this should no longer happen. The root issue is that the library should not throw a different parsing exception when it encounters an error response from the service that it can’t parse, instead it should return the error response verbatim which is what it does in 12.13.

[JoshLove-msft]

Seems like the StorageRequestFailedDetailsParser is not catching exceptions that are thrown if there is an issue with the content - https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Common/src/Shared/StorageRequestFailedDetailsParser.cs#L52

If no parser is specified, we do catch exceptions that occur while parsing and return the content verbatim - https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/src/RequestFailedException.cs#L265-L269.

We can look into doing the same for Storage.