Unauthorized error when authenticated with UserAssignedManagedIdentity on Us Gov Cloud

Report

After helm install from the official chart and from after this PR: https://github.com/promitor/charts/pull/36

I encounter the Operation returned an invalid status code 'Unauthorized' error while trying to query resources from AzureGraphResource after Authenticated.

Note: I have checked the identity binding (with Reader and Contributor roles of the subscription) with azure-cli image and I could az login --identity and fetch the any resources in the subscription.

Expected Behavior

After authenticated with the UserAssignedManagedIdentity, the client could query for the resources defined. In this case, I use the sample:

resourceDiscoveryGroups:
- name: container-registry-landscape
  type: ContainerRegistry

Actual Behavior

Got the below error log and the service discovery returned 500 HTTP code.

Steps to Reproduce the Problem

1. Create an AKS cluster on Azure Gov cloud subscription and enable the pod identity add-on.
2. Create the Azure User Assigned Identity, add the role assignment of Contributor, Reader and Monitoring Reader for it to the subscription.
3. Configure the Service Discovery with the label aadpodidbinding=<identity-name-on-k8s>
4. After the pod is running healthy, call the API health check with includeDendencies=true

kubectl port-forward -n default svc/promitor-agent-resource-discovery 8889
curl "http://localhost:8889/api/v1/health?includeDependencies=true"

1. Got the following result:

{
    "entries": {
        "azure-resource-graph": {
            "data": {
                "Subscription 4d47ec28-XXXX-XXXX-XXXX-XXXXXXXXXXXX": {
                    "isSuccessful": false,
                    "message": "Unable to discover resources due to an unexpected exception: Operation returned an invalid status code 'Unauthorized'"
                }
            },
            "description": "One or more subscriptions are unhealthy.",
            "duration": "00:00:01.0022206",
            "status": "Unhealthy",
            "tags": []
        }
    },
    "status": "Unhealthy",
    "totalDuration": "00:00:01.0037660"
}

Component

Resource Discovery

Version

0.4.0

Configuration

Configuration for Helm chart:

azureLandscape:
  cloud: "UsGov"
  tenantId: 5766f879-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  subscriptions:
  - 4d47ec28-XXXX-XXXX-XXXX-XXXXXXXXXXXX
azureAuthentication: 
  identity:
    binding: aad-promitor
    id: c5d63849-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  mode: UserAssignedManagedIdentity
image:
  repository: "ghcr.io/tomkerkhove/promitor-agent-resource-discovery"
  tag: "0.4.0"

resourceDiscoveryGroups:
- name: container-registry-landscape
  type: ContainerRegistry

resource-discovery-declaration.yaml configmap:

version: v1
azureLandscape:
  tenantId: 5766f879-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  subscriptions:
  - 4d47ec28-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  cloud: UsGov
resourceDiscoveryGroups:
- name: container-registry-landscape
  type: ContainerRegistry

runtime.yaml

server:
  httpPort: "88"
authentication:
  mode: UserAssignedManagedIdentity
  identityId: c5d63849-XXXX-XXXX-XXXX-XXXXXXXXXXXX
cache:
  enabled: "true"
  durationInMinutes: "5"
telemetry:
  applicationInsights:
    isEnabled: "false"
  containerLogs:
    isEnabled: "true"
  defaultVerbosity: "Error"

Logs

Unable to discover resources due to an unexpected exception: Operation returned an invalid status code 'Unauthorized'
Microsoft.Azure.Management.ResourceGraph.Models.ErrorResponseException: Operation returned an invalid status code 'Unauthorized'
at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClient.ResourcesWithHttpMessagesAsync(QueryRequest query, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClientExtensions.ResourcesAsync(IResourceGraphClient operations, QueryRequest query, CancellationToken cancellationToken)
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.<>c__DisplayClass13_0.<<QueryAsync>b__0>d.MoveNext() in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 80
--- End of stack trace from previous location where exception was thrown ---
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.<>c__DisplayClass14_0`1.<<InteractWithAzureResourceGraphAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.InteractWithAzureResourceGraphAsync[TResponse](String queryName, String query, List`1 targetSubscriptions, Func`2 interactionFunc) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 93
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.QueryAsync(String queryName, String query, List`1 targetSubscriptions) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 77
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.QueryForResourcesAsync(String queryName, String query, List`1 targetSubscriptions) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 67
at Promitor.Agents.ResourceDiscovery.Health.AzureResourceGraphHealthCheck.ProbeSubscriptionAsync(String query, String subscription) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraphHealthCheck.cs:line 84

Platform

Microsoft Azure

Contact Details

[e-mail]

[tomkerkhove]

Thanks. It could be that or permissions, but I’d expect an unauthorized there then.

It might be this: https://github.com/tomkerkhove/promitor/pull/1648/files#diff-630f87ab9d90eb937b2970b338be8e942706f90d9c248a618b158a23618d84f1R113

[tomkerkhove]

Azure Monitor uses https://github.com/tomkerkhove/promitor/pull/1648/files#diff-630f87ab9d90eb937b2970b338be8e942706f90d9c248a618b158a23618d84f1R116 but thath one looks OK though.