AWS metadata and STS API calls are slow

During the configuration of the AWS provider, it calls into multiple AWS APIs to check various endpoints and identity metadata. Times vary quite a bit depending on the speed of your network, however it’s not uncommon for these to add up to 10-15 seconds of lag before an update even begins running.

Here are the specific calls:

• https://github.com/hashicorp/aws-sdk-go-base/blob/91fe5671bc1bd585d180bba8954ef56d48bef22b/awsauth.go#L270-L294
• https://github.com/hashicorp/aws-sdk-go-base/blob/91fe5671bc1bd585d180bba8954ef56d48bef22b/session.go#L104-L109
• https://github.com/hashicorp/aws-sdk-go-base/blob/91fe5671bc1bd585d180bba8954ef56d48bef22b/session.go#L131-L139
• https://github.com/hashicorp/aws-sdk-go-base/blob/91fe5671bc1bd585d180bba8954ef56d48bef22b/session.go#L141-L158

Note that you can set config variables to skip this logic:

• pulumi config set aws:skipCredentialsValidation true
• pulumi config set aws:skipMetadataApiCheck true
• pulumi config set aws:skipRequestingAccountId true

For some reason, the provider seems to call these APIs twice, one of which ignores the config. @lukehoban posited that this could be due to the way we do a prepass over configuration to validate and check for defaults. If so, that seems like it’s a bug that we do it without having first applied the configuration.

Note also that you can set AWS_METADATA_TIMEOUT=0 which shortens the timeouts of the AWS metadata API calls and does have a small noticeable effect.

I don’t know precisely what to do here, but we could consider setting our own defaults differently than the underlying Terraform provider. I don’t know enough about what those APIs are doing – it appears, for instance, that the metadata API check is determining whether the update is happening from within an AWS data center (though why the code needs to know that, I’m not quite sure).

[uLan08]

I encountered an issue somewhat related to this. I was trying to spin up the basic example from pulumi new aws-javascript but pulumi up was just hanging indefinitely. The most I waited was probably 5 mins.

Setting pulumi config set aws:skipCredentialsValidation true fixed it for me,

Here are some of my environment info in case you need them.

▶ node -v
v10.16.3

▶ pulumi version
v1.10.1

"@pulumi/pulumi": "^1.0.0",
"@pulumi/aws": "^1.0.0",
"@pulumi/awsx": "^0.18.10"


config:
  aws:profile: foo
  aws:region: ap-southeast-1
  pulumi:template: aws-javascript

It was stuck at

▶ pulumi up --logtostderr -v=9 2> out.txt
Previewing update (dev):
     Type                 Name             Plan        
     pulumi:pulumi:Stack  pulumi-demo-dev  create..

Last entries from logs:

I0212 22:33:41.285702   73852 eventsink.go:60] AWS Auth provider used: "SharedCredentialsProvider"
I0212 22:33:41.285742   73852 eventsink.go:63] eventSink::Debug(<{%reset%}>AWS Auth provider used: "SharedCredentialsProvider"<{%reset%}>)
I0212 22:33:41.288928   73852 eventsink.go:60] Trying to get account information via sts:GetCallerIdentity
I0212 22:33:41.288964   73852 eventsink.go:63] eventSink::Debug(<{%reset%}>Trying to get account information via sts:GetCallerIdentity<{%reset%}>)

[shousper]

Sorry @stack72, doesn’t look like this is my issue. It ramped up, so I dug deeper… looks like I’m in this trap: https://github.com/hashicorp/terraform/issues/27350 https://github.com/golang/go/issues/42700

😞