(CDK Pipelines) Deploy stage, vpc.prepare failed due to S3 Access Denied

I have a working code pipeline using CDK python. Recently someone else in the team did some change just move some of the code to different folder. then the code pipeline stopped working, failed at the first step in deploying stage, vpc.Prepare. the reason was S3 Access Denied.

My first thinking is to revert the change and try again, and I did it. it did not help.

Then I removed all the stacks in cloud formation, started over. it did not help.

I found it was due to S3 Access Denied. so I change the S3 bucket pipelinestack-mappipelineartifactsbucket822fb6ba-1cjz3xh6o8l9j to public access, and allow public read to the artifacts within this bucket. Then the Prepare succeeded.

To verify, I turned off the public access of the s3, then it stopped working, stuck at the Prepare step. Turning it on again, then it is in progress again.

in the cloud formation, I can see that the deployment role policy is administrator access, which should not have this kind access issue, correct? But why after granting public access, it worked?

Keep the bucket publicly accessible is not ideal.

Please help to solve the issue.

Thanks

Reproduction Steps

What did you expect to happen?

Prepare step should not be stuck

What actually happened?

Prepare step got stuck when code pipeline is deploy stage

Environment

• **CDK CLI Version : 1.111.0 (build 556ca93)

• Framework Version:

• **Node.js Version:**v16.2.0

• **OS : big sure

• Language (Version): Python 3.8.2

Other

This is the error message:

Error message
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: K4T79835ANWTDQY3; S3 Extended Request ID: Hw8uwS1+GrSvsbkem0Xh+XdEDzwdmr2u4yv7Szun9SSbgrtmbw8uwNLq+z1rinQFdqf0xpUiLic=; Proxy: null)

Could you track this request ID to figure out what is happening under the hood?

---

This is 🐛 Bug Report

[devendhirababu]

Hi @peterwoodworth,

Thanks for re-opening this issue, I would like to inform you that the issue was resolved yesterday with the help of an AWS support team(Hammad R) we had multiple calls, after spending one week of time.

He analyzed the error message deeply and found that issues with an Bootstrap bucket policy. We added the permission in the bootstrap bucket policy allowing cfn-exe-role, deployment-role, pipeline-role (Principle) and “Action”:s3* for the resources (Artifact bucket).

He saved me from the embarrassing situation, I really thankful to Hammad R (AWS Support) and aws-cdk-GitHub Team.

Regards, Deva

[jy-metservice]

ame the S3 permission denied error

@jy-metservice How did you do this ? I have added ACL, and also a policy like this but it is still failing

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::artifact-bucket-acc",
                "arn:aws:s3:::artifact-bucket-acc/*"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::acc:role/cdk-hnb659fds-cfn-exec-role-acc-region"
                }
            }
        }
    ]
}

@leantorres73 I used the following ACL. (Note: I have altered our account ID and bucket name). I had to give access to both the cfn-exec-role + the cfn-deploy role

{
    "Version": "2012-10-17",
    "Id": "AccessControl",
    "Statement": [
        {
            "Sid": "CDKToolkitAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-cfn-exec-role-123456789-us-east-1",
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-deploy-role-123456789-us-east-1"
                ]
            },
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:PutObject*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::our-bucket-name",
                "arn:aws:s3:::our-bucket-name/*"
            ]
        }
    ]
}