pipelines: Generated OverflowPolicy exceeds policy limit

Describe the bug

I use cdk pipeline construct to create a pipeline that deploys stacks to 22 regions in two accounts (dev and prod). I tested my pipeline with few regions and it worked fined. When scaling to all 22 regions the pipeline construct generates 21 policies attached to the pipeline role. The limit of 20 managed policy can not be increased. The pipeline fails in the self update step with an error:

 0/26 | 4:00:27 PM | CREATE_FAILED        | AWS::IAM::ManagedPolicy     | xxx/Pipeline/Role/OverflowPolicy21 (xxxPipelineRoleOverflowPolicy2134C9B563) Cannot exceed quota for PoliciesPerRole: 20 (Service: AmazonIdentityManagement; Status Code: 409; Error Code: LimitExceeded; Request ID: 78a11314-xxxx-494c-9303-0760303fd8c9; Proxy: null)

Additionally running cdk synth locally will produce a warning:

 Policy too large: 21 exceeds the maximum of 20 managed policies attached to a Role

Expected Behavior

I would expect the pipeline to generate one policy per target deployment account and not exceed the policy limit.

It seems like there is a lot of waste in the generated policy document. When I look into the template I see a repetition of these three statement blocks:

      {
       "Action": [
        "s3:GetObject*",
        "s3:GetBucket*",
        "s3:List*",
        "s3:DeleteObject*",
        "s3:PutObject",
        "s3:PutObjectLegalHold",
        "s3:PutObjectRetention",
        "s3:PutObjectTagging",
        "s3:PutObjectVersionTagging",
        "s3:Abort*"
       ],
       "Effect": "Allow",
       "Resource": [ 
          "SOME_ARN_TO_S3_REPLICATION_BUCKET",
          "SOME_ARN_TO_S3_REPLICATION_BUCKET/*" 
        ]
      },
      {
       "Action": [
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:Encrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*"
       ],
       "Effect": "Allow",
       "Resource": "*"
      },
      {
       "Action": "sts:AssumeRole",
       "Effect": "Allow",
       "Resource": { "ARN_TO_CROSS_ACCOUNT_ROLE"}
      }

These statements are repeated for every replication bucket in each target region. Because the Action block is identical in every case it would be great to only extend the Resource block by combining the resources.

Current Behavior

With the recent PR 75bfce7 (Role policies cannot grow beyond 10k) the construct splits the policy document into multiple role policies which I see in the asset:

The number of policies generated scales with the number of cross-region stacks and accounts. In my case, the pipeline generated 21 policies for 22 regions for only single account (dev), it will grow additionally if I add prod account to my pipeline.

Reproduction Steps

Create a list of account/regions combination that is greater than 21:

const ACCOUNTS = {
  Beta: 'xxxxxx'
};

const stageWaves = {
  Beta: {
    waves: [
      [
        {region: REGIONS.EU_CENTRAL_1, account: ACCOUNTS.Beta}
      ],
      [
        {region: REGIONS.EU_WEST_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_NORTHEAST_1, account: ACCOUNTS.Beta},
        {region: REGIONS.US_EAST_1, account: ACCOUNTS.Beta},
      ],
      [
        {region: REGIONS.US_WEST_1, account: ACCOUNTS.Beta},
        {region: REGIONS.EU_NORTH_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_SOUTH_1, account: ACCOUNTS.Beta},
        {region: REGIONS.SA_EAST_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_SOUTHEAST_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_NORTHEAST_2, account: ACCOUNTS.Beta},
        {region: REGIONS.EU_WEST_2, account: ACCOUNTS.Beta},
        {region: REGIONS.US_EAST_2, account: ACCOUNTS.Beta},
        {region: REGIONS.US_WEST_2, account: ACCOUNTS.Beta},
        {region: REGIONS.EU_WEST_3, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_SOUTHEAST_2, account: ACCOUNTS.Beta},
        {region: REGIONS.CA_CENTRAL_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_NORTHEAST_3, account: ACCOUNTS.Beta},
        {region: REGIONS.EU_SOUTH_1, account: ACCOUNTS.Beta},
        {region: REGIONS.ME_SOUTH_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_EAST_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AF_SOUTH_1, account: ACCOUNTS.Beta},
        {region: REGIONS.AP_SOUTHEAST_3, account: ACCOUNTS.Beta}
      ],
    ],
  },
}

create a simple pipeline using code commit as a source:

const pipeline = new CodePipeline(this, "Pipeline", {
      crossAccountKeys: true,
      synth: new ShellStep('Synth', {
        input: CodePipelineSource.codeCommit(
          Repository.fromRepositoryName(this, 'SourceRepo', 'YOUR_CODECOMMIT_REPO'), 'main'),
        commands: [
          'npm i',
          'npm run build',
          'npx cdk synth',
        ],
      }),
      selfMutation: true,
      dockerEnabledForSynth: true,
      codeBuildDefaults: {
        buildEnvironment: {
          computeType: ComputeType.MEDIUM,
        },
      }
    });

Create stages and waves for each region:

    stageWaves.Beta.waves.forEach((wave, index) => {
        const appWave = pipeline.addWave(`App-Beta-${index}`);
        wave.forEach(env => {
          appWave.addStage(new AppStage(this, `AppStage-${env.region}`, {env: env}))
        });
      }
    )

The appStage can be a simple stack that creates at least one regional resource:

class AppStage extends Stage {
  constructor(scope: Construct, id: string, props?: StageProps) {
    super(scope, id, props);
    new AppStack(this, 'AppStack');
  }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.26.0 (build a409d63)

Framework Version

No response

Node.js Version

v16.14.0

OS

macOS Big Sur 11.6.6.

Language

Typescript

Language Version

3.9.7

Other information

No response

[rix0rrr]

Two questions:

• Did this use to work properly for 22 regions and did a recent change break this, or is this the first time you are trying this?
• Can you try adding{ "context": { "@aws-cdk/aws-iam:minimizePolicies": true }} to your cdk.json and seeing if that makes a difference?

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.