Java SDK does not support EKS IAM for service accounts

EKS IAM Service Account Role introduces a new environment variable “AWS_WEB_IDENTITY_TOKEN_FILE” and based on the documentation on these two pages, the Java SDK should use “AWS_WEB_IDENTITY_TOKEN_FILE” for credentials if exists.

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

I have tried the latest Java SDK 2.9.19 and it doesn’t seem to work. There is a similar discussion on DotNet SDK at here:

https://github.com/aws/aws-sdk-net/issues/1413

I couldn’t find docs saying “AWS_WEB_IDENTITY_TOKEN_FILE” is in the list of credentials chain for SDK Java 2. I am wondering if this is implemented or not.

Expected Behavior

Based on the EKS doc link above, Java SDK should recognize the environment variable “AWS_WEB_IDENTITY_TOKEN_FILE” and use it to call AssumeRoleWithWebIdentity for access/secret/session tokens.

Current Behavior

I am getting Access Denied with Java SDK in a correctly setup EKS pod with service account. In the same pod, I was able to run aws s3 ls and it worked, which means the token is correct.

Exception in thread "main" software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 467C66AAEAD354A9)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleErrorResponse(HandleResponseStage.java:115)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleResponse(HandleResponseStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:58)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.s3.DefaultS3Client.listBuckets(DefaultS3Client.java:2035)
        at com.example.s3.S3BucketOps.main(S3BucketOps.java:63)

Steps to Reproduce (for bugs)

The code I am using to test is from sample S3 code at here: https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3BucketOps.java

I set up a pod in EKS with service account and IAM role properly setup. In the pod, I used aws cli to test the token and it worked. However, the Java SDK didn’t work.

Your Environment

• AWS Java SDK version used: 2.9.19
• JDK version used: 1.8
• Operating System and version: Linux running from a container (pod on EKS)

[starchx]

@endre-synnes The issue is due to WebIdentityTokenFileCredentialsProvider is not in the default credentials provider chain

The workaround for now is to specify the provider in the client initialization:

import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
...
S3Client s3 = S3Client.builder()
                              .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
                              .region(region)
                              .build();

Looking forward that WebIdentityTokenFileCredentialsProvider to be added to the default credentials provider chain.

[thereisnospoon]

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.