[BUG] DefaultAzureCredential and ImdsManagedIdentitySource - bad 404 handling

Library name and version

Azure.Identity 1.6.0 and 1.5.0

Describe the bug

In ImdsManagedIdentitySource, if the metadata server returns a status different of 400 / 502 / 504, it is assumed a success.

If the service listening on 169.254.169.254 is not Azure MetaData service, the response returned is a 404 which is afterwards treated as a “success” and parsed as JSON.

This “success” treatment generates a complete authentication failure when using the DefaultAzureCredential(), instead of continuing and trying other authentication mechanisms as it is an “uncaught exception” there.

Expected behavior

The 404 error should be treated as a CredentialUnavailableError in order to continue the authentication process and try the next authentication methods.

Some Invalid Json handling shall also be implemented

Actual behavior

DefaultAzureCredentials stops after this exception and do not try further authentication mechanisms.

The exception generated is the following:

Unhandled exception. Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Managed Identity response was not in the expected format. See the inner exception for details.
Status: 404 (Not Found)

Content:


Headers:
Date: Fri, 15 Apr 2022 06:20:39 GMT
Server: EC2ws
Connection: close
Content-Type: text/html
Content-Length: 339

See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
 ---> Azure.RequestFailedException: Managed Identity response was not in the expected format. See the inner exception for details.
Status: 404 (Not Found)

Content:


Headers:
Date: Fri, 15 Apr 2022 06:20:39 GMT
Server: EC2ws
Connection: close
Content-Type: text/html
Content-Length: 339

 ---> System.Text.Json.JsonReaderException: '<' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
   at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, JsonReaderOptions readerOptions, MetadataDb& database, StackRowStack& stack)
   at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedBytes)
   at System.Text.Json.JsonDocument.Parse(Stream utf8Json, JsonDocumentOptions options)
   at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
   at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)

Issue has been confirmed (and mitigated “harshly”) by preventing the access to the metadata server :

netsh advfirewall firewall add rule name="BUGMITIGATION" dir=out action=block protocol=tcp localip=any remoteip=169.254.169.254

With this mitigation we were able to again use az cli authentication with DefaultAzureCredentials().

Reproduction Steps

The following code triggers the authentication error when http://169.254.169.254/metadata/identity/oauth2/token returns a 404 page (tested on aws workspaces)

using System;
using Azure.Identity;
using Microsoft.Rest;
namespace debugauth
{
    class Program
    {
        static void Main(string[] args)
        {
            var defaultClient = new DefaultAzureCredential();
            var token = defaultClient.GetToken(new Azure.Core.TokenRequestContext(new[] { $"https://management.azure.com/.default" }));
            ServiceClientCredentials serviceClientCreds = new TokenCredentials(token.Token);
            Console.WriteLine("Hello World!");
        }
    }
}

Environment

• hosting platform : aws workspaces
• dotnet info :

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.407
 Commit:    1a1785612e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17763
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.407\

Host (useful for support):
  Version: 5.0.16
  Commit:  1016676966

.NET SDKs installed:
  5.0.407 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

[darkoa-msft]

@christothes why was this issue closed? This is still a bug - Identity library expects JSON but, I am assuming, IMDS returns an XML on 404. We are observing this in Azure, with Managed Identity deployed. I understand that the root cause is somewhere else, but the library should handle this kind of failure.

[BC89]

Following these official doc fresh from MS - https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview-for-developers?tabs=portal%2Cdotnet and it bombs locally. VS2022 - Docker Container. That’s pretty poor.