Minimum TTL on instance profile credentials results in unstable connections to AWS resources

Describe the bug

Seems like commit 3f67bf74 introduced a minimum wait of 15 minutes before credentials are refreshed if credentials received from instance metadata expire in less than 15 minutes. This poses a problem as tokens received from instance metadata don’t refresh when you request them, but refresh asynchronously in the background. When the token TTL is equal to this 15-minute refresh interval an app using the SDK will inevitably get into a state similar to the following:

1. 00:00 AWS magic refreshes instance metadata credentials
2. 00:03 The SDK fetches the token, sees it expires in less than 15 minutes, and schedules a refresh in 15 minutes
3. 00:10 AWS magic refreshes instance metadata credentials
4. 00:15 The SDK’s token expires, all of my calls to DynamoDB are now failing ☹️
5. 00:18 The SDK fetches the token, sees it expires in less than 15 minutes, and schedules a refresh in 15 minutes
6. …repeat forever!

The impact is more pronounced if the SDK fetches a token closer to expiry (eg token expires at 00:15 and is fetched at 00:14).

This boils down to differences between v2.17.158’s HttpCredentialsProvider.getPrefetchTime(...) which scheduled tokens to be refreshed immediately if they expire in less than 15 minutes and v2.17.159’s InstanceProfileCredentialsProvider.getPrefetchTime(...) which has the behavior described above. ☝️

Expected behavior

If instance metadata tokens are expired they should be refreshed immediately, if tokens expire in less than 15 minutes they should refresh before they expire

Current behavior

Instance metadata tokens which are expired or nearly expired are refreshed in 15 minutes even if they expire in less than 15 minutes

Steps to Reproduce

Deploy an SDK v2.17.159 app to an EC2 instance whose tokens’ TTLs are the 15 minute minimum and try to use instance metadata credentials to access DynamoDB

Possible Solution

Return the old behavior of immediately refreshing expired tokens Schedule soon-to-expire tokens to refresh before they expire

Context

No response

AWS Java SDK version used

v2.17.159

JDK version used

11

Operating System and version

Base image openjdk:11-jdk-slim

[millems]

Ahh, cool. Thanks for taking the time to dig in and confirm! The release going out today should fix this problem.

[millems]

That sounds very scary. Let me revert that feature in the next release until we can determine what’s going on.