question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Missing dependency leads to opaque failover with DefaultCredentialsProvider

See original GitHub issue

Describe the issue

When using the DefaultCredentialsProvider with the intent to have WebIdentityTokenFileCredentialsProvider provide the credentials in a production kubernetes (EKS) environment without having the sts library on the classpath the DefaultCredentialsProvider silently fails over to the next CredentialsProvider in the chain.

The missing sts dependency leads to a ClassNotFoundException being rethrown as an IllegalStateException (in WebIdentityCredentialsUtils.factory()), but is immediately caught and assigned to a field. When the AwsCredentialsProviderChain later calls resolveCredentials on the WebIdentityTokenFileCredentialsProvider that exception is rethrown and immediately being caught again by AwsCredentialsProviderChain, moving to the next CredentialsProvider in the chain.

In my case the next successful provider in the chain was the instance profile of the underlying worker node, discovered via ec2 metadata service, not having the required policies attached to call the API in question - in this case s3 - leading to a 403 Forbidden, putting myself on a wild goose chase trying to find out why my (perfectly well configured) kubernetes service account doesn’t seem to have the required permission to call s3.

In my opinion the ClassNotFoundException should probably not be suppressed, as when reaching that point it’s rather obvious (given that two environment variables or properties with the correct names exist) that it’s the developer’s intention to use web identity and the missing library being a mistake of said developer rather than a intended fallback to another CredentialProvider.

Steps to Reproduce

import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.regions.Region;

public class App {
    public static void main(String[] args) {
         var s3Client = S3Client.builder().region(Region.EU_WEST_1).build();
         var request = GetObjectRequest.builder().bucket("my-bucket").key("/my-key").build();
         s3Client.getObject(request); // Exception
    }
}

For simplicity’s sake have a configured aws-cli profile (with no permissions to GetObject from my-bucket) lying in your home directory (~/.aws). Run with s3 library (and transitional dependencies, but not sts) on classpath and both of these environment variables set to some string:

  • AWS_WEB_IDENTITY_TOKEN_FILE
  • AWS_ROLE_ARN

Your Environment

  • AWS Java SDK version used: 2.13.41
  • JDK version used: 11

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
aschejacommented, Nov 26, 2020

Sure.

0reactions
aschejacommented, Dec 3, 2020

Thanks!

Read more comments on GitHub >

github_iconTop Results From Across the Web

No results found

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found