question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

AzureML: User managed identity attached to compute cluster does not work

See original GitHub issue
  • Package Name: azure-identity
  • Package Version: 1.5.0
  • Operating System: Linux
  • Python Version: 3.7

Describe the bug

A custom docker container running on a compute cluster with a user managed identity does not seem to authenticate with Azure identity when running in AzureML

This is the logs I when I run my python code:

WARNING 2020-12-21 18:04:30,166 azure.identity._internal.get_token_mixin AppServiceCredential.get_token failed: No token received.
WARNING 2020-12-21 18:04:30,166 azure.identity._internal.decorators ManagedIdentityCredential.get_token failed: No token received.
WARNING 2020-12-21 18:04:30,166 azure.identity._credentials.chained ChainedTokenCredential.get_token failed: ManagedIdentityCredential raised unexpected error "No token received."
WARNING 2020-12-21 18:04:30,166 azure.identity._credentials.chained ChainedTokenCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	ManagedIdentityCredential: No token received.

To Reproduce Steps to reproduce the behavior:

  1. Create a compute cluster with a managed identiy attached to read data from a blob storage
  2. Create a custom docker image with the azure sdk to fetch data from blob storage
  3. Run the aforementioned docker image in the AzureML cluster created above.

Expected behavior Python SDK can authenticate using a manged identity.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:23 (12 by maintainers)

github_iconTop GitHub Comments

1reaction
gdippolitocommented, Jan 4, 2021

Hi @mccoyp thanks for your response.

Yes I have used a custom docker images based on nvidia/cuda runtime image.

I have managed to fix the issue by doing the following things:

  1. Rollback azure-identity to 1.4.1 (due to https://github.com/Azure/azure-sdk-for-python/issues/15361)
  2. Authenticate with identity.ManagedIdentityCredential(client_id='MY_CLIENT_ID) (MY_CLIENT_ID is the client id of the managed identity attached to the compute cluster in the workspace). Without setting the client_id the authentication fails.

Does this make sense to you?

Thanks

0reactions
raz4commented, May 18, 2021

Ok understood. I get warnings and eventually a clear authentication error like below…

WARNING:azure.identity._internal.get_token_mixin:AppServiceCredential.get_token failed: No token received.
WARNING:azure.identity._internal.decorators:ManagedIdentityCredential.get_token failed: No token received.
WARNING:azure.identity._credentials.chained:DefaultAzureCredential.get_token failed: ManagedIdentityCredential raised unexpected error "No token received."
WARNING:azure.identity._credentials.chained:DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
	ManagedIdentityCredential: No token received.
...
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.

The warnings are similar to what the original issue described, so that’s how I ended up here. Anyway, thanks for your help!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Set up service authentication - Azure Machine Learning
The Azure ML compute cluster uses a managed identity to retrieve connection information for datastores from Azure Key Vault and to pull Docker ......
Read more >
No more passwords, please!. How Azure Machine Learning ...
The best way to assign a managed identity to a Compute Cluster is to specify the managed identity when the Compute Cluster is...
Read more >
Manage access to an Azure Machine Learning workspace
NOTE: "az ml workspace share" command does not work for federated account by Azure Active ... User-assigned managed identity with Azure ML compute...
Read more >
azurerm_machine_learning_com...
At the moment, there is also no possibility to specify ssh User Account ... Managed Identity IDs to be assigned to this Machine...
Read more >
Exam DP-100 - NET
You can also configure managed identities for use with Azure Machine Learning compute cluster. This managed identity is independent of ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found