InvalidIdentityToken : Token file expired, refresh token

Describe the bug

On long live application like a Worker or an API. After a long period the app is not authenticated anymore and receive the error InvalidIdentityToken :

{
  "message": "Couldn't retrieve verification key from your identity provider,  please reference AssumeRoleWithWebIdentity documentation for requirements",
  "path": [
    "createAffectation"
  ],
  "stack": "InvalidIdentityToken: Couldn't retrieve verification key from your identity provider,  please reference AssumeRoleWithWebIdentity documentation for requirements\n
at deserializeAws_queryAssumeRoleWithWebIdentityCommandError (/app/node_modules/@aws-sdk/client-sts/dist/cjs/protocols/Aws_query.js:363:41)\n
at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n
at async /app/node_modules/@aws-sdk/middleware-serde/dist/cjs/deserializerMiddleware.js:6:20\n
at async StandardRetryStrategy.retry (/app/node_modules/@aws-sdk/middleware-retry/dist/cjs/StandardRetryStrategy.js:51:46)\n
at async /app/node_modules/@aws-sdk/middleware-logger/dist/cjs/loggerMiddleware.js:6:22\n
at async /app/node_modules/@aws-sdk/client-sts/dist/cjs/defaultStsRoleAssumers.js:70:33\n
at async SignatureV4.signRequest (/app/node_modules/@aws-sdk/client-sqs/node_modules/@aws-sdk/signature-v4/dist/cjs/SignatureV4.js:84:29)\n
at async /app/node_modules/@aws-sdk/client-sqs/node_modules/@aws-sdk/middleware-signing/dist/cjs/middleware.js:14:22\n    
at async StandardRetryStrategy.retry (/app/node_modules/@aws-sdk/client-sqs/node_modules/@aws-sdk/middleware-retry/dist/cjs/StandardRetryStrategy.js:51:46)\n
at async /app/node_modules/@aws-sdk/middleware-sdk-sqs/dist/cjs/send-message.js:6:18\n

Your environment

SDK version number

"@aws-sdk/client-s3": "^3.18.0"
"@aws-sdk/client-sqs": "^3.23.0"
"@aws-sdk/client-sts": "^3.18.0"

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

14.15.4

Steps to reproduce

getConfigFromEnv() {
  return {
      credentials: fromTokenFile({
        webIdentityTokenFile: env.aws.tokenFile,
        roleArn: env.aws.roleArn,
        roleSessionName: env.aws.sessionName,
        durationSeconds: env.aws.sessionDuration,
        roleAssumerWithWebIdentity: getDefaultRoleAssumerWithWebIdentity(),
      }),
    };
}

export abstract class SQSBaseClient {
  public sqs: SQS;
  public queueUrl: string;
  private initPromise: Promise<void> | undefined;
  protected fifoQueue: boolean;
  public messageSizeLimit: number;

  protected constructor(public queueName: string, config?: SQSClientConfig) {
    this.sqs = new SQS({
      ...config,
      ...getConfigFromEnv(),
      region: env.aws.region,
      apiVersion: "2012-11-05",
    });
    this.fifoQueue = queueName.endsWith(".fifo"); //FIFO queues always end in .fifo by AWS rules
  }

This is the abstraction to use SQS, we do not want to handle the authentification directly.

Observed behavior

Our Backend is running in an EKS cluster, Kubernetes is injecting a token in a pod to assume a role but this token has an expiration. The token is changed on the machine before the token expires.

After a long period the sdk is not able to authenticate to AWS using the past token to assume a role.

Expected behavior

The token in the file is re read when the past token is expired. I do not see any documentation in the sdk v3 nodejs to do it.

In the v2 I can do it : https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/WebIdentityCredentials.html

How can we change the credentials in the sdk ?

[carlosescura]

We are still experiencing the same issues days after the outage, but this time they are just sporadic on some EC2 nodes. Anyone still having this issue?

We’re on us-east-1

[tiagoernst]

Any news regarding this issue ?