"Error validating token: IDX10223" Whilst getting a KeyVault secret

• Package Name: azure-keyvault-secrets
• Package Version: 4.1.0
• Operating System: Ubuntu 20.04
• Python Version: 3.8

Describe the bug After some period of time, running the following will return an error

credential = DefaultAzureCredential()
self._secret_client = SecretClient(vault_url=keyvault_url, credential=credential)
# ... after a undetermined amount of time
self._secret_client.get_secret(secret_name)

Stacktrace

HttpResponseError: (Unauthorized) Error validating token: IDX10223
  File "a/server.py", line 48, in readyz
    cache.get_secret('a-readyz')
  File "a/cache.py", line 21, in get_secret
    secret = self.storage.get_secret(normalised_secret_name, secret_name)
  File "a/storage.py", line 50, in get_secret
    secret_value_obj = self._secret_client.get_secret(secret_name)
  File "/usr/local/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/azure/keyvault/secrets/_client.py", line 66, in get_secret
    bundle = self._client.get_secret(
  File "/usr/local/lib/python3.8/site-packages/azure/keyvault/secrets/_shared/_generated/v7_0/operations/_key_vault_client_operations.py", line 1625, in get_secret
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/usr/local/lib/python3.8/site-packages/azure/core/exceptions.py", line 102, in map_error
    raise error

To Reproduce

Unknown, it happens after a few hours/days, I’ve had this happen in 3 different environments now,

Expected behavior I expected it to return a secret

Additional context

This is running in a container, on Kubernetes in Azure using pod-aad-identity to assign a managed identity to the worker the container is running on, and then handles calls to the metadata address.

[chlowell]

You could enable debug logging on the client with logging_enable=True:

self._secret_client = SecretClient(vault_url=keyvault_url, credential=credential, logging_enable=True)

That would log all its requests at DEBUG level with nothing redacted. So, security warning there, but it would let you read the token from a failed request’s Authorization header and check its exp claim (the token is a JWT). Alternatively, you could do similar with the credential instead, i.e. DefaultAzureCredential(logging_enable=True). Then you’d see the token request itself:

Request URL: 'http://169.254.169.254/...'
Request method: 'GET'
Request headers:
    'Metadata': 'true'
    'User-Agent': 'azsdk-python-identity/1.3.1 ...'
Request body:
None
Response status: 200
Response headers:
    'Content-Type': 'application/json; charset=utf-8'
    'Date': 'Mon, 20 Jul 2020 16:19:39 GMT'
    'Content-Length': '1643'
Response content:
{"access_token":"...","expires_on":"1595347982", ...}

For a monkeypatch approach, SecretClient adds tokens to requests here and the credential deserializes the token here.

[msftbot[bot]]

Hi, we’re sending this friendly reminder because we haven’t heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don’t hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!