CredentialsProviderChain unable to load credentials on EKS

Describe the bug

When loading a Spring application on an AWS EKS cluster (which is able load apps credentials through the Java SDK), the Kotlin SDK’s CredentialsProviderChain fails to load them through its chain.

No credentials could be loaded from the chain: CredentialsProviderChain -> EnvironmentCredentialsProvider -> ProfileCredentialsProvider -> StsWebIdentityProvider -> EcsCredentialsProvider -> ImdsCredentialsProvider

Expected behavior

I would expect the Kotlin SDK to be able to fetch the same credentials that the Java SDK is able to load

Current behavior

I have instrumented my Spring Configuration file to log some information to help.

This is the Spring Configuration file:

@Configuration
class DynamoDBConfig(private val logger: Logger = LoggerFactory.getLogger(DynamoDBConfig::class.java)) {

    @Bean
    fun amazonDynamoDB(): DynamoDbClient = runBlocking {
        DynamoDbClient.fromEnvironment().apply {
            debug()
        }
    }

    private suspend fun debug() {
        kotlin.runCatching {
            logger.info("Java SDK Credential Provider Chain - Initializing")
            DefaultAWSCredentialsProviderChain.getInstance().let {
                logger.info("Java SDK Credential Provider Chain - Getting Credentials from ProviderChain")
                logger.info("Java SDK Credential Provider Chain - Credentials: ${it.credentials}")
            }
        }.onFailure { ex -> logger.error("Java SDK Credential Provider Chain - Exception", ex) }

        kotlin.runCatching {
            logger.info("Kotlin SDK Credential Provider Chain - Initializing")
            DefaultChainCredentialsProvider().let {
                logger.info("Kotlin SDK  Credential Provider Chain - Getting Credentials from ProviderChain")
                logger.info("Kotlin SDK  Credential Provider Chain - Credentials: ${it.getCredentials()}")
            }
        }.onFailure { ex -> logger.error("Kotlin SDK Credential Provider Chain - Exception", ex) }
    }
}

These are the logs (I have removed timestamps, log level and noisy stuff to make it more readable)

The logs from the Java CredentialsProviderChain

DynamoDBConfig message=Java SDK Credential Provider Chain - Initializing
DynamoDBConfig message=Java SDK Credential Provider Chain - Getting Credentials from ProviderChain
com.amazonaws.auth.AWSCredentialsProviderChain message=Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
com.amazonaws.auth.AWSCredentialsProviderChain message=Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)
com.amazonaws.auth.AWSCredentialsProviderChain message=Unable to load credentials from WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName
com.amazonaws.auth.AWSCredentialsProviderChain message=Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@404ced67: profile file cannot be null
com.amazonaws.auth.AWSCredentialsProviderChain message=Loading credentials from com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@71531dd7
DynamoDBConfig message=Java SDK Credential Provider Chain - Credentials: com.amazonaws.auth.BasicSessionCredentials@20256a0b

The logs from the Kotlin CredentialsProviderChain

DynamoDBConfig message=Kotlin SDK Credential Provider Chain - Initializing
DynamoDBConfig message=Kotlin SDK  Credential Provider Chain - Getting Credentials from ProviderChain
aws.sdk.kotlin.runtime.auth.credentials.CredentialsProviderChain message=unable to load credentials from aws.sdk.kotlin.runtime.auth.credentials.EnvironmentCredentialsProvider@79ca7bea: Missing value for environment variable `AWS_ACCESS_KEY_ID`
aws.sdk.kotlin.runtime.auth.credentials.ProfileCredentialsProvider message=Loading credentials from profile `default`
aws.sdk.kotlin.runtime.auth.credentials.CredentialsProviderChain message=unable to load credentials from aws.sdk.kotlin.runtime.auth.credentials.ProfileCredentialsProvider@3869a6e5: could not find source profile default
aws.sdk.kotlin.runtime.auth.credentials.CredentialsProviderChain message=unable to load credentials from aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@d25e878: Required field `roleArn` could not be automatically inferred for StsWebIdentityCredentialsProvider. Either explicitly pass a value, set the environment variable `AWS_ROLE_ARN`, or set the JVM system property `aws.roleArn`
aws.sdk.kotlin.runtime.auth.credentials.CredentialsProviderChain message=unable to load credentials from aws.sdk.kotlin.runtime.auth.credentials.EcsCredentialsProvider@578c3fd9: Container credentials URI not set
aws.sdk.kotlin.runtime.auth.credentials.CredentialsProviderChain message=unable to load credentials from aws.sdk.kotlin.runtime.auth.credentials.ImdsCredentialsProvider@49491770: failed to load instance profile
DynamoDBConfig message=Kotlin SDK Credential Provider Chain - Exception exception=aws.sdk.kotlin.runtime.auth.credentials.CredentialsProviderException: No credentials could be loaded from the chain: CredentialsProviderChain -> EnvironmentCredentialsProvider -> ProfileCredentialsProvider -> StsWebIdentityProvider -> EcsCredentialsProvider -> ImdsCredentialsProvider

Steps to Reproduce

Try to get Credentials from an application running on an EKS Cluster on which the Java SDK is able to load IAM profiles etc.

Possible Solution

It looks like the Kotlin SDK is missing a EC2ContainerCredentialsProvider maybe?

Context

I am ready to try a new DynamoDB persistence to replace a 5TB relational database that is out of whack. It’s a mission critical system where I work and this is blocking us.

In the meantime, I will try use a delegation pattern to the Java SDK credentials provider chain 🤞

AWS Kotlin SDK version used

0.15.0, 0.16.0, 0.16.1-beta

Platform (JVM/JS/Native)

JVM

Operating System and version

Docker, Java 11.0.14.1

[lucix-aws]

Thanks for the stack trace - we can see the underlying exception there: EC2MetadataError: Request forbidden: IMDS is disabled or the caller has insufficient permissions.

While I try to figure out exactly what’s happening here, in the interest of trying to unblock you, I believe you’re missing the session token when pulling in the credentials from Java.

[vgiguere]

Reporting Back in case someone else has similar issue and finds this thread.

It looks like the source of the issue is the KUBE2IAM pod, which proxies all STS calls on our cluster. The version we use (0.10.7) does not support IMDSv2 properly.

I will try to the version bumped to 0.11.0 and report back