[CLI] cdk deploy doesn't work after force upgrading to the new bootstrap version

I upgraded to cdk 1.61 and got asked to upgrade my bootstrap version from 3 to 4, which at first failed as described here https://github.com/aws/aws-cdk/issues/10016, only I didn’t wait for the fix and force upgraded.

If I run cdk diff I get the following output (with out the changes I made, only the ones relating to the bootstrap):

Stack wm
IAM Statement Changes
┌───┬──────────────────────────────────────────┬────────┬──────────────────────────────────────────┬────────────────────────────────────────────┬───────────┐
│   │ Resource                                 │ Effect │ Action                                   │ Principal                                  │ Condition │
├───┼──────────────────────────────────────────┼────────┼──────────────────────────────────────────┼────────────────────────────────────────────┼───────────┤
│ - │ {"Fn::ImportValue":"CdkBootstrap-hnb659f │ Allow  │ kms:Decrypt                              │ AWS:${Custom::CDKBucketDeployment8693BB649 │           │
│   │ ds-FileAssetKeyArn"}                     │        │ kms:DescribeKey                          │ 68944B69AAFB0CC9EB8756C/ServiceRole}       │           │
└───┴──────────────────────────────────────────┴────────┴──────────────────────────────────────────┴────────────────────────────────────────────┴───────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Parameters
[+] Parameter BootstrapVersion BootstrapVersion: {"Type":"AWS::SSM::Parameter::Value<String>","Default":"/cdk-bootstrap/hnb659fds/version","Description":"Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store."}

Resources
[~] AWS::IAM::Policy Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C/ServiceRole/DefaultPolicy CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRoleDefaultPolicy88902FDF 
 └─ [~] PolicyDocument
     └─ [~] .Statement:
         └─ @@ -35,16 +35,6 @@
            [ ] },
            [ ] {
            [ ]   "Action": [
            [-]     "kms:Decrypt",
            [-]     "kms:DescribeKey"
            [-]   ],
            [-]   "Effect": "Allow",
            [-]   "Resource": {
            [-]     "Fn::ImportValue": "CdkBootstrap-hnb659fds-FileAssetKeyArn"
            [-]   }
            [-] },
            [-] {
            [-]   "Action": [
            [ ]     "s3:GetObject*",
            [ ]     "s3:GetBucket*",
            [ ]     "s3:List*",
[~] AWS::Lambda::Function LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8a LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aFD4BFC8A 
 └─ [~] Metadata
     ├─ [-] Removed: .aws:asset:path
     └─ [-] Removed: .aws:asset:property

Other Changes
[+] Unknown Rules: {"CheckBootstrapVersion":{"Assertions":[{"Assert":{"Fn::Not":[{"Fn::Contains":[["1","2","3"],{"Ref":"BootstrapVersion"}]}]},"AssertDescription":"CDK bootstrap stack version 4 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."}]}}

when I run cdk deploy I get the following error message:

 ❌  wm failed: Error [ValidationError]: AccessDenied. User doesn't have permission to call ssm:GetParameters
    at Request.extractError (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ValidationError',
  time: 2020-08-28T15:20:02.851Z,
  requestId: '629a101e-a63a-43c7-8f45-dc70bbeba284',
  statusCode: 400,
  retryable: false,
  retryDelay: 743.7164270441746
}
AccessDenied. User doesn't have permission to call ssm:GetParameters

Environment

• **CLI Version : 1.16.1
• **Framework Version: 1.16.1
• **Node.js Version: v12.18.3
• **OS : ubuntu 20.4
• **Language (Version): ts

This is 🐛 Bug Report

[shivlaks]

you might want to try running bootstrap again (without the force and with cloudformation-execution-policies created)

Your CdkToolkit (or whatever you named your bootstrap stack) should have a resource called CloudFormationExecutionRole - can you check the AdministratorAccess policy is attached to it?

I’m guessing it’s empty. We should probably add validation here to prevent users from getting into this scenario.

You can run bootstrap again to get that updated

cdk bootstrap --context @aws-cdk/core:newStyleStackSynthesis=1 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://accountid/region

Your cdk deploy should work after that. let me know how it goes!

[jonny-rimek]

My CloudFormationExecutionRole did indeed not have any policy attached to it running cdk bootstrap --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess fixed it.

thanks for your help