[pipelines] UpdatePipeline stage of new empty pipeline fails due to missing permissions (ref. #9227)

I’m attempting to set up an initial blank pipeline via Java11, and have the same permissions issue noted in issue #9227, where the UpdatePipeline stage fails to due missing CloudFormation permissions needed for cdk deploy.

Reproduction Steps

Deploy empty (no application stages) pipeline from https://github.com/HtyCorp/cdk-repro-1

Requires: Java 11 (tested: Corretto-11.0.8.10.1), Maven (tested: 3.6.0), must have a Secrets Manager secret “GitHubRepoAccessToken” in account containing GitHub OAuth token for CodePipeline.

git clone https://github.com/HtyCorp/cdk-repro-1.git && cd cdk-repro-1 && mvn clean install && cdk synth && cdk deploy --require-approval=never

What did you expect to happen?

Default pipeline stages (Source -> Build -> UpdatePipeline) of deployed pipeline all execute successfully since no deployments or changes are required.

Role generated by CdkPipeline construct has permission to invoke CloudFormation, or is used by UpdatePipeline to assume account’s ‘cdk--deploy-role-’ role for CloudFormation permissions.

What actually happened?

Pipeline is successfully created but UpdatePipeline stage is unable to execute cdk deploy due to missing cloudformation:GetTemplate permission:

User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee

Detailed CodeBuild logs:

[Container] 2020/08/11 14:25:11  BUILD: 1 commands
[Container] 2020/08/11 14:25:11 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2020/08/11 14:25:11 Phase context status code:  Message: 
[Container] 2020/08/11 14:25:11 Entering phase INSTALL
[Container] 2020/08/11 14:25:11 Running command npm install -g aws-cdk
/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk
+ aws-cdk@1.57.0
added 216 packages from 186 contributors in 7.207s

[Container] 2020/08/11 14:25:25 Phase complete: INSTALL State: SUCCEEDED
[Container] 2020/08/11 14:25:25 Phase context status code:  Message: 
[Container] 2020/08/11 14:25:25 Entering phase PRE_BUILD
[Container] 2020/08/11 14:25:25 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2020/08/11 14:25:25 Phase context status code:  Message: 
[Container] 2020/08/11 14:25:25 Entering phase BUILD
[Container] 2020/08/11 14:25:25 Running command cdk -a . deploy CdkPipeline --require-approval=never --verbose
CDK toolkit version: 1.57.0 (build 2ccfc50)
Command line arguments: { _: [ 'deploy' ],
  a: '.',
  app: '.',
  'require-approval': 'never',
  requireApproval: 'never',
  verbose: 1,
  v: 1,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  fail: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  '$0': '/usr/local/bin/cdk',
  STACKS: [ 'CdkPipeline' ],
  stacks: [ 'CdkPipeline' ] }
merged settings: { versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: '.',
  context: {},
  tags: [],
  assetMetadata: true,
  requireApproval: 'never',
  toolkitBucket: {},
  staging: true }
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to ap-southeast-2
Resolving default credentials
Looking up default account ID from STS
Default account ID: 111122223333
Setting "CDK_DEFAULT_ACCOUNT" environment variable to 111122223333
context: { 'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true }
--app points to a cloud assembly, so we bypass synth
CdkPipeline: deploying...
Waiting for stack CDKToolkit to finish creating or updating...
CdkPipeline: checking if we can skip deploy
Call failed: getTemplate({"StackName":"CdkPipeline","TemplateStage":"Original"}) => User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee

 ❌  CdkPipeline failed: AccessDenied: User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee
User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee
AccessDenied: User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee
    at Request.extractError (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

[Container] 2020/08/11 14:25:27 Command did not exit successfully cdk -a . deploy CdkPipeline --require-approval=never --verbose exit status 1
[Container] 2020/08/11 14:25:27 Phase complete: BUILD State: FAILED
[Container] 2020/08/11 14:25:27 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk -a . deploy CdkPipeline --require-approval=never --verbose. Reason: exit status 1
[Container] 2020/08/11 14:25:27 Entering phase POST_BUILD
[Container] 2020/08/11 14:25:27 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2020/08/11 14:25:27 Phase context status code:  Message: 

Policy of generated IAM role CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE (not modified in CDK app):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:ap-southeast-2:111122223333:log-group:/aws/codebuild/CDKDeploymentPipeline-selfupdate",
                "arn:aws:logs:ap-southeast-2:111122223333:log-group:/aws/codebuild/CDKDeploymentPipeline-selfupdate:*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "codebuild:CreateReportGroup",
                "codebuild:CreateReport",
                "codebuild:UpdateReport",
                "codebuild:BatchPutTestCases"
            ],
            "Resource": "arn:aws:codebuild:ap-southeast-2:111122223333:report-group/CDKDeploymentPipeline-selfupdate-*",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:*:iam::*:role/*-deploy-role-*",
                "arn:*:iam::*:role/*-publishing-role-*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": "cloudformation:DescribeStacks",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "s3:ListBucket",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::cdkpipeline-deploymentpipelineartifactsbucketf7fe-abcd12341a7n9",
                "arn:aws:s3:::cdkpipeline-deploymentpipelineartifactsbucketf7fe-abcd12341a7n9/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:ap-southeast-2:111122223333:key/abcd1234-e992-436c-b98d-4dbdea13b3df",
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*"
            ],
            "Resource": "arn:aws:kms:ap-southeast-2:111122223333:key/abcd1234-e992-436c-b98d-4dbdea13b3df",
            "Effect": "Allow"
        }
    ]
}

Environment

• CLI Version : 1.57.0 (build 2ccfc50)
• Framework Version: 1.57.0
• Node.js Version: v10.22.0
• OS : Ubuntu 18.04.4 LTS
• Language (Version): Java (11)

Other

---

This is 🐛 Bug Report

[MamishIo]

Edit: Turns out this was my fault: cdk.json was misformatted, so new-style stack synthesis wasn’t enabled. After correcting that, the UpdatePipeline cdk deploy command is correctly using the account’s bootstrapped deployment role for the previously failing actions.

For posterity, fixed by changing:

{
  "app": "mvn -e -q exec:java",
  "output": "target/cloud-assembly",
  "@aws-cdk/core:newStyleStackSynthesis": "true"
}

to:

{
  "app": "mvn -e -q exec:java",
  "output": "target/cloud-assembly",
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": true
  }
}

[matwerber1]

@cbrgm yes, actually found my problem.

The issue was that I was correctly updating cdk.json with the "@aws-cdk/core:newStyleStackSynthesis": true flag, but the mistake was that I was locally running cdk deploy afterward to test the changes.

Being new to CDK pipelines, I didn’t realize this was the mistake. With the way the pipeline runs, its actually going to pull the latest changes from your source code repository when it gets to the relevant build scripts - and since I had not committed and pushed the change to cdk.json to remote, it was still building based on the old cdk.json that was missing the new stack synthesis flag.

Lesson learned for me is that, with CDK pipelines, after your very first local cdk deploy to get the pipeline up and running, you should rely on pushing commits to the repo for future changes.

Hopefully this helps you?