[BUG] ManagedIdentityCredential credential leads to slow requests with AAD Workload Identity

Describe the bug

I am trying out AAD Workload Identity. I have written a sample application that uses ManagedIdentityCredential and prints every X seconds the size of a blob in a storage account. The first access is fast, but then the Azure SDK complains about an expired AAD credential and takes ~30s to get a new one.

Exception or Stack Trace

Env var PERIOD: 5
Env var BLOB_NAME: <blob name>
Env var ENDPOINT: https://<storageaccount>.blob.core.windows.net/
Env var CONTAINER_NAME: <container>
Env var AZURE_FEDERATED_TOKEN_FILE: <token path>
Contents of AZURE_FEDERATED_TOKEN_FILE 
---
<REDACTED>
---
2022-08-25T11:50:17.243358Z Start: Creating credential
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2022-08-25 13:50:17.467 [main] [INFO] com.azure.core.implementation.jackson.JacksonVersion - Package versions: jackson-core=2.13.3, jackson-databind=2.13.3, jackson-dataformat-xml=2.13.3, jackson-datatype-jsr310=2.13.3, azure-core=1.31.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
2022-08-25 13:50:18.069 [main] [DEBUG] com.azure.identity.ManagedIdentityCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID
2022-08-25T11:50:18.069820Z Finish: Creating credential
2022-08-25T11:50:18.070354Z Start: Creating blobServiceClient
2022-08-25 13:50:18.186 [main] [DEBUG] com.azure.core.implementation.http.HttpClientProviders - Using com.azure.core.http.netty.NettyAsyncHttpClientProvider as the default HttpClientProvider.
2022-08-25T11:50:18.665058Z Finish: Creating blobServiceClient
2022-08-25T11:50:18.665722Z Start: Creating blobContainerClient
2022-08-25T11:50:18.670823Z Finish: Creating blobContainerClient
2022-08-25T11:50:18.671427Z Start: Creating blobClient
2022-08-25T11:50:18.687736Z Finish: Creating blobClient
2022-08-25T11:50:18.688170Z Start: Get properties
2022-08-25 13:50:20.363 [parallel-1] [INFO] com.azure.identity.ManagedIdentityCredential - Azure Identity => Managed Identity environment: AZURE AKS TOKEN EXCHANGE
2022-08-25 13:50:20.363 [parallel-1] [INFO] com.azure.identity.ManagedIdentityCredential - Azure Identity => getToken() result for scopes [https://storage.azure.com/.default]: SUCCESS
2022-08-25 13:50:20.363 [parallel-1] [INFO] com.azure.core.implementation.AccessTokenCache - Acquired a new access token.
2022-08-25T11:50:21.123163Z Stop: Get properties
2022-08-25T11:50:21.123561Z Size: 14024704
2022-08-25T11:50:26.130085Z Start: Get properties
2022-08-25 13:50:50.569 [parallel-2] [INFO] com.azure.identity.ManagedIdentityCredential - Azure Identity => Managed Identity environment: AZURE AKS TOKEN EXCHANGE
2022-08-25 13:50:50.569 [parallel-2] [INFO] com.azure.identity.ManagedIdentityCredential - Azure Identity => getToken() result for scopes [https://storage.azure.com/.default]: SUCCESS
2022-08-25 13:50:50.569 [parallel-2] [INFO] com.azure.core.implementation.AccessTokenCache - Acquired a new access token at 1661424627 seconds after expiry. Retry may be attempted after 30 seconds.
2022-08-25T11:50:50.620474Z Stop: Get properties
2022-08-25T11:50:50.620579Z Size: 14024704

The strange expiration time is approximately now - 3600 -30

To Reproduce

Create an AAD Workload Identity setup as described on the project homepage. Create a storage account, a container and a blob. Give the AAD application read permissions to the blob.

Try to print the blob size while using a ManagedIdentityCredential created via

var credential = new ManagedIdentityCredentialBuilder().build();

Expected behavior Repeated request during the validation period of the AAD token (3600 seconds) are fast (RTT and a bit). New AAD tokens are only requested every hour or so.

Setup (please complete the following information):

• AKS 1.22
• Java 11
• azure sdk according to bom version 1.2.5
• AAD workload identity

Additional context

When using ClientAssertionCredential, I can connect and read the properties of the blob, authentication is quick and new Azure AD tokens are fetched every 55 minutes.

2022-08-25T17:17:10.770495Z Start: Get properties
2022-08-25T17:17:10.786232Z Stop: Get properties
2022-08-25T17:17:10.786413Z Size: 10223616
2022-08-25T17:18:10.786571Z Start: Get properties
2022-08-25T17:18:10.803340Z Stop: Get properties
2022-08-25T17:18:10.803408Z Size: 10223616
2022-08-25T17:19:10.803564Z Start: Get properties
2022-08-25T17:19:10.820161Z Stop: Get properties
2022-08-25T17:19:10.820244Z Size: 10223616
2022-08-25T17:20:10.820446Z Start: Get properties
Contents of AZURE_FEDERATED_TOKEN_FILE: < REDACTED >
2022-08-25 17:20:11.298 [Thread-15] [INFO] com.azure.identity.ClientAssertionCredential - Azure Identity => getToken() result for scopes [https://storage.azure.com/.default]: SUCCESS
2022-08-25 17:20:11.299 [Thread-15] [INFO] com.azure.core.implementation.AccessTokenCache - Acquired a new access token at 297 seconds before expiry. Retry may be attempted after 30 seconds. The token currently cached will be used.
2022-08-25T17:20:11.314743Z Stop: Get properties
2022-08-25T17:20:11.314812Z Size: 10223616
2022-08-25T17:21:11.315069Z Start: Get properties
2022-08-25T17:21:11.331762Z Stop: Get properties
2022-08-25T17:21:11.331843Z Size: 10223616

May be related to #30043

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[mariusx]

Hi! We seems to have the same issue and this is a blocker right now.

com.azure.identity.implementation.MSIToken is always expired when response from AAD token endpoint is parsed. When using workload identity the access token returned from AAD has the format documented here:

https://github.com/uglide/azure-content/blob/master/articles/active-directory/active-directory-v2-protocols-oauth-code.md#successful-response-1

{
    "access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik5HVEZ2ZEstZnl0aEV1Q...",
    "token_type": "Bearer",
    "expires_in": 3599,
    "scope": "https%3A%2F%2Fgraph.microsoft.com%2Fmail.read",
    "refresh_token": "AwABAAAAvPM1KaPlrEqdFSBzjqfTGAMxZGUTdM0t4B4...",
    "id_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIyZDRkMTFhMi1mODE0LTQ2YTctOD...",
}

Parsing and instantiation of a com.azure.identity.implementation.MSIToken will set the field expiresAt to EPOC + 3599, which is equal to “1970-01-01T00:59:59Z”

Adding to com.azure.identity.implementation.MSIToken.parseDateToEpochSeconds a fix could be to verify if the parameter dateTime is less than Instant.now().getEpochSecond()

 private static Long parseDateToEpochSeconds(String dateTime) {
    try {
      long secondsToAdd = Long.parseLong(dateTime);
      // expiresIn is expected to be in seconds
      // Can be offset from now or seconds since EPOCH
      long secondsSinceEpoc = Instant.now().getEpochSecond();
      return secondsSinceEpoc > secondsToAdd && MAX_SECONDS_TO_EXPIRY <= secondsToAdd ? secondsSinceEpoc + secondsToAdd : secondsToAdd;
    } catch (NumberFormatException e) {
      LOGGER.verbose(e.getMessage());
    }
    ...

[g2vinay]

@mariusx I am investigating this issue. Aiming to have a fix/resolution out this week, will update here.