[BUG] ManagedIdentityCredential attempts to parse a non-success response

Library name and version

Microsoft.Azure.WebJobs.Extensions.Storage.Queues 5.0.0

Describe the bug

ManagedIdentityCredential authentication sometimes fails directly after a deployment of an Azure Function. I put additional details in https://github.com/Azure/azure-functions-host/issues/8623.

Exception while executing function: StartCreateHostUsage ManagedIdentityCredential authentication failed: Managed Identity response was not in the expected format. See the inner exception for details.
Status: 403 (Forbidden)



Headers:
Date: Tue, 09 Aug 2022 23:00:04 GMT
Server: Kestrel
X-CORRELATION-ID: REDACTED
Content-Length: 0



See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot Managed Identity response was not in the expected format. See the inner exception for details.
Status: 403 (Forbidden)



Headers:
Date: Tue, 09 Aug 2022 23:00:04 GMT
Server: Kestrel
X-CORRELATION-ID: REDACTED
Content-Length: 0
The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0.

Expected behavior

A better error message if the managed identity endpoint responds with a 403. It would be nice if it retried or a retry policy could be set up. Is there a way for me to wait for the system managed identity to be ready during startup?

Actual behavior

I seems like the managed identity is not quite ready yet in the App Service environment and a 403 response is returned. Even with a bad response, the code still tries to parse the response body and fails with a JsonReaderException.

Reproduction Steps

Deploy an Azure Function that uses system managed identity and try to write to a storage queue. Here is the code we are using.

type StartCreateUsage() =
    [<FunctionName(nameof StartCreateUsage)>]
    member x.Run
        // Hourly process to create usage
        ([<TimerTrigger("1 0 * * * *", RunOnStartup = false, UseMonitor = true)>] timer: TimerInfo, [<Queue(StorageQueues.createUsage)>] createUsageQueue: ICollector<CreateUsage.Params>)
        (log: ILogger): Task<unit> =
            task {
                let usageHours = getUsageHours (timer.ScheduleStatus.Last.ToUniversalTime()) DateTime.UtcNow
                if usageHours.Length > 1 then
                    let usageHours = usageHours |> List.map (fun d -> d.SortableString)
                    log.Warningf "creating usage for more than one hour: %A" usageHours
                for usageHour in usageHours do
                   CreateUsage.Params(UsageHour = usageHour) |> createUsageQueue.Add
            }

Environment

It is dotnet 6. The base docker image is: FROM mcr.microsoft.com/azure-functions/dotnet:4.0.1.16816-dotnet6-appservice

[jsquire]

Hi @cataggar. Thank you for reaching out and we regret that you’re experiencing difficulties. There’s nothing that the credential can do to address the root problem, as the endpoint itself is returning a 403. (your Functions issue is a good path forward there). I’m not sure why we’re attempting to parse the body of a non-success response - that seems like it may be a bug. I’ve updated the title to reclassify and have routed this to the team member best able to assist.

[cataggar]

@schaabs, the service isn’t supposed to send a 403 error, but was. Here is the internal IcM that is now resolved. It probably shouldn’t be retriable. Yes, a better error message for non-success status codes is what this issue is about.