Raise NoCredentialsError when no credentials are found

This topic has come up before, see issues #29 and #1403. I’m re-raising this, with some details below. I have also created a PR to fix this #1752.

Certain credential providers, such as the InstanceMetadataProvider have the chance to fail to load credentials on start, even when the credentials exist and valid. Typically, such failures would get handled by the RefreshableCredentials class; however, that refresh only occurs if the provider was able to find credentials (thereby being the chosen credential provider).

The above situation results in a scenario where it’s possible to create a session that does not have credentials. This will result in the client raising a NoCredentialsError upon the first usage of the credentials. That error will not have the ability to auto-resolve.

Having systematic credential failures like this are critical to be found upon session creation, rather than client use, as it enables services utilizing the session to know with certainty whether the session is usable or not.

To expand on this further:

• The current implementation is different from the Java SDK. In Java an exception is thrown when no credentials are found in the chain. See this.
• This has the potential for a web service to start, thinking it is healthy, pass a health check, and yet be unable to access any AWS services. A potential workaround is to call session.get_credentials() and verify that it is not None; however, this places an unnecessary burden on the user, since that workaround would need to be implemented by all endpoints.
• Similar to the above - any background workers will also have the same issue. That problem becomes further complicated, because there could be multiple calls, by the same application, to the instance metadata service where some calls passed and others didn’t.
• Imagine a containerized implementation, such as Kubernetes, where there is the potential to have mutiple containers running on the same node, where some are able to reach the instance store and others are not. This is worsened when there any many containers running on one node (think lots of lightweight services), especially upon initial deploy, where the thundering herd problem may become an issue.
• For anyone expecting a provider that will refresh credentials, this will never occur, because no instance of RefreshableCredentials will be created (the credentials will be None).

[bhtucker]

This is still an issue. I wonder if it’s exaggerated on EKS, where the additional network hops may eat into an otherwise-reasonable timeout.

For anyone reading the above, the env vars are AWS_METADATA_SERVICE_TIMEOUT and AWS_METADATA_SERVICE_NUM_ATTEMPTS, and these settings do apply to the case where IMDS returns a 200 with an empty body (ie, botocore does see that as worth retrying).

[attilasimon72]

We have just worked around this bug by setting:

AWS_METADATA_SERVICE_TIMEOUT=65 (or any reasonably high value) AWS_METADATA_SERVICE_NUM_ATTEMPTS=1

Due to #2304, this config:

AWS_METADATA_SERVICE_TIMEOUT=1 AWS_METADATA_SERVICE_NUM_ATTEMPTS=15

is practically equivalent with:

AWS_METADATA_SERVICE_TIMEOUT=1 AWS_METADATA_SERVICE_NUM_ATTEMPTS=1

(If the IMDS query runs on ReadTimeoutError then botocore ignores AWS_METADATA_SERVICE_NUM_ATTEMPTS and breaks the retry loop on the first try.)

If the IMDS query ends with timeout then botocore will cache the empty credential as if it was valid. Setting AWS_METADATA_SERVICE_TIMEOUT to a high value reduces the chance of getting/caching an empty cred.