(bootstrap): Bootstrapping with CDK v1.139.0 removes IAM policy from the ECR asset repository

What is the problem?

Bootstrapping from 1.138.0 to v1.139.0 removes IAM policy from the cdk-hnb659fds-container-assets-1234567890-eu-central-1 ECR repository resulting in Lambdas being unable to run due to not able to access their images. The following policy was removed:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "LambdaECRImageRetrievalPolicy",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": [
        "ecr:BatchGetImage",
        "ecr:DeleteRepositoryPolicy",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:SetRepositoryPolicy"
      ],
      "Condition": {
        "StringLike": {
          "aws:sourceArn": "arn:aws:lambda:eu-central-1:1234567890:function:*"
        }
      }
    }
  ]
}

resulting in the following Lambda error:

Failed to restore the function MyLambda: The function does not have permission to access the specified image.

Reproduction Steps

1. Bootstrap account with v1.138.0
2. Add permission policy (or just deploy a Docker Lambda which will add a policy automatically)
3. Bootstrap account with v1.139.0 and check the IAM policy gets removed from ECR cdk-hnb659fds-container-assets-1234567890-eu-central-1 repository

What did you expect to happen?

Bootstrap process should NOT remove IAM policy from ECR cdk-hnb659fds-container-assets-1234567890-eu-central-1 repository

What actually happened?

Bootstrap process removed IAM policy from ECR cdk-hnb659fds-container-assets-1234567890-eu-central-1 repository

CDK CLI Version

1.139.0

Framework Version

No response

Node.js Version

v14.15.0

OS

Ubuntu 20.04

Language

Python

Language Version

No response

Other information

No response

[polothy]

@NGL321 hello - we ran into this today and found the source of the problem. See this lambda doc.

Lambda needs ECR to have a policy on it to allow it to pull images. The “funny” thing that lambda does though is automatically add the policy to ECR:

If the Amazon ECR repository does not include these permissions, Lambda adds ecr:BatchGetImage and ecr:GetDownloadUrlForLayer to the container image repository permissions. Lambda can add these permissions only if the Principal calling Lambda has ecr:getRepositoryPolicy and ecr:setRepositoryPolicy permissions.

The problem is, on CDK bootstrap upgrade, the ECR policy that lambda added can be removed, thus breaking things for lambda (it cannot pull).

The fix seems to be to add this policy to ECR in the CDK bootstrap.

[scedward]

We also saw this unexpected change after upgrading to the latest bootstrap stack. For a quick fix, making a trivial change to the Dockerfile of the affected Lambda was enough to change the asset hash and led to restoration of the lost policy statement.