(custom-resources): AwsCustomResource leaks assumed role to other custom resources

The runtime handler of the AwsCustomResource does not correctly reset the credentials after executing when given a assumedRoleArn in any of the AwsSdkCall objects.

This means if you have two AwsCustomResource constructs in the same stack, and the first one that is deployed supplies a assumedRoleArn then the second one will fail to deploy if it executes any commands that are not covered by the policy of the assumed role of the first custom resource.

This obviously only happens if the execution context of the Lambda is reused, which is quite likely if you have a dependency between the custom resources so they’re not executed concurrently.

Reproduction Steps

import * as cdk from '@aws-cdk/core'
import * as iam from '@aws-cdk/aws-iam'
import * as customResources from '@aws-cdk/custom-resources'

export class TestStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    const fooRole = new iam.Role(this, 'FooRole', {
      assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com')
    })
    fooRole.addToPolicy(
      new iam.PolicyStatement({
        actions: ['s3:CreateBucket'],
        resources: ['arn:aws:s3:::Foo']
      })
    )

    // We specify an explicit role that is only allowed to create a bucket called 'Foo'
    new customResources.AwsCustomResource(this, 'Foo', {
      onCreate: {
        service: 'S3',
        action: 'createBucket',
        parameters: {
          Bucket: 'Foo'
        },
        assumedRoleArn: fooRole.roleArn,
        physicalResourceId: customResources.PhysicalResourceId.fromResponse('Bucket.Id')
      },
      installLatestAwsSdk: false,
      policy: customResources.AwsCustomResourcePolicy.fromSdkCalls({
        resources: customResources.AwsCustomResourcePolicy.ANY_RESOURCE
      })
    })

    // We do not explicitly assume a role and expect the custom resource policy to grant permission to CreateBucket
    new customResources.AwsCustomResource(this, 'Bar', {
      onCreate: {
        service: 'S3',
        action: 'createBucket',
        parameters: {
          Bucket: 'Bar'
        },
        physicalResourceId: customResources.PhysicalResourceId.fromResponse('Bucket.Id')
      },
      installLatestAwsSdk: false,
      policy: customResources.AwsCustomResourcePolicy.fromSdkCalls({
        resources: customResources.AwsCustomResourcePolicy.ANY_RESOURCE
      })
    })
  }
}

What did you expect to happen?

Deployment of both resources should succeed.

What actually happened?

The deployment of the second custom resource fails due to insufficient permissions of the role that was specified for the first custom resource.

Environment

• CDK CLI Version : 1.109.0
• Framework Version: 1.109.0
• Node.js Version: v15.14.0
• OS : MacOS 11.2.3
• Language (Version): TypeScript (3.9.7)

Other

The issue is located here https://github.com/aws/aws-cdk/blob/master/packages/@aws-cdk/custom-resources/lib/aws-custom-resource/runtime/index.ts#L134. I propose to replace lines 134-143 with the following:

let credentials: AWS.Credentials | undefined

if (call.assumedRoleArn) {
  const timestamp = new Date().getTime()

  const params = {
    RoleArn: call.assumedRoleArn,
    RoleSessionName: `${physicalResourceId}-${timestamp}`
  }

  credentials = new AWS.ChainableTemporaryCredentials({
    params: params
  })
}

const awsService = new (AWS as any)[call.service]({
  apiVersion: call.apiVersion,
  region: call.region,
  credentials: credentials
})

Instead of modifying the global AWS SDK config, we only apply it to the temporary service client.

---

This is 🐛 Bug Report

[theipster]

Ultimately, the way I see the situation is quite simply: the Lambda was designed to be a singleton, therefore it should ideally be made stateless or else there’ll always be a risk of prior side effects affecting the output - as seen in all of the scenarios mentioned above.

Unfortunately, in its current form, the Lambda isn’t stateless. State exists in the form of AWS.config.credentials which, once set (bear in mind, some invocations may not necessarily attempt to set it at all, and some invocations may attempt to set an existing value to another value), persists throughout the lifetime of that Lambda execution context and thus means it leaks into all subsequent invocations that happen to execute in the same context.

While the workaround using nested stacks might appear to work,

1. It loses the efficiency gained from using the singleton pattern. In other words, an application would have multiple Lambdas in deployment simultaneously (each incurring their own costs, etc.), all of which are actually just duplicates of each other but simply holding different state at runtime.
2. It forces re-architecting the application to use nested stacks, which introduces additional complexity that may or may not be warranted.
3. It still doesn’t solve the problem when the same custom resource needs to assume a different role (or indeed, none at all) between its create/update/delete lifecycle hooks (e.g. @mattvanstone’s use case), because there’s no way to apply the same technique of splitting the hooks of a single custom resource across separate stacks.

To summarise, making the Lambda stateless will solve the problem while keeping true to the original design.

[theipster]

I’ve been trialling @nicolai-shape’s suggestion above for roughly a week now, with great success. 👍

While the fix itself seems pretty straightforward, I guess the complication is how to test it. I’ve prepared some failing tests to describe the scenario, which then starts passing once the above fix gets applied: theipster/aws-cdk#1.

Does this look useful / worthy of submitting a formal PR?