[CLI] cdk deploy keeps calling describeStackEvents() for ~10 minutes after stack update

AWS CDK cdk deploy keeps calling describeStackEvents() for about 10 minutes after the CloudFormation stack has been successfully updated. It seems that CDK iterates through the entire event history of the stack in 5 second intervals. This makes the cdk deploy command take a very long time.

During the delay AWS CDK appears to be stuck in the “creating CloudFormation changeset” phase. In reality it has already created the changeset and applied it to the CloudFormation stack.

Reproduction Steps

You might need a CloudFormation stack that has a lot of event history to notice that AWS CDK iterates through all of them, causing the long delay.

You can see the describeStackEvents() iteration in debug output by using cdk deploy -v -v -v. It will keep showing entries like this (with different NextToken as it progresses):

[AWS cloudformation 200 5.499s 0 retries] describeStackEvents({
  StackName: 'webcat-registrar-dev',
  NextToken: 'f73Uze7bI/CdCn4qrXZ31hZlkhi5jbWWog...'
}) 

What did you expect to happen?

cdk deploy should finish in about 3 minutes.

What actually happened?

cdk deploy takes 14 minutes to complete.

Environment

• CLI Version : 1.56.0
• Framework Version: 1.56.0
• Node.js Version: v12.18.2
• OS : Linux (Ubuntu in Docker)
• Language (Version): TypeScript 3.9.7

Other

---

This is 🐛 Bug Report

[rix0rrr]

Yep. Turns out we are accidentally quadratic!

[kennu]

I double checked by adding a console.log() of every stack event that AWS CDK scans through. It really iterates all the stack events from today all the way back to 2019-04-07 when the stack was created.

Unless I misunderstand something, the whole logic of StackActivityMonitor.readEvents() is flawed. It will always scan through the entire stack event history, which keeps growing forever and making deployments slower and slower.

This is the last entry scanned before cdk deploy completes after 15 minutes of iterating events:

{
  StackId: 'arn:aws:cloudformation:eu-west-1:xxx:stack/webcat-registrar-dev/xxx',
  EventId: 'xxx',
  StackName: 'webcat-registrar-dev',
  LogicalResourceId: 'webcat-registrar-dev',
  PhysicalResourceId: 'arn:aws:cloudformation:eu-west-1:xxx:stack/webcat-registrar-dev/xxx',
  ResourceType: 'AWS::CloudFormation::Stack',
  Timestamp: 2019-04-07T00:12:50.615Z,
  ResourceStatus: 'REVIEW_IN_PROGRESS',
  ResourceStatusReason: 'User Initiated'
}
Stack webcat-registrar-dev has completed updating