question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(aws-cdk): --no-rollback not working as expected

See original GitHub issue

First, thanks for the great PR for the disable rollback feature. I was testing this in my environment and noticed it’s not working as expected and this could be an issue related to cloudformation.

To dive deep into the root cause, I noticed the CDK API will always create change set and then execute the change set with --disable-rollback option, however, execute-change-set --disable-rollback will actually not stop the rollback. And we can reproduce this with AWS CLI so I believe it’s not related to CDK.

Reproduction Steps

  1. create a CDK app with bad configuration
cdk synth > template.yaml
  1. create change set with AWS CLI
aws cloudformation create-change-set \ 
    --stack-name my-application \
    --change-set-name my-change-set \
    --template-body file://template.yaml \
    --capabilities CAPABILITY_IAM \
    --change-set-type CREATE
  1. execute the change set with the --disable-rollback option.
aws cloudformation execute-change-set --change-set-name my-change-set  --disable-rollback
  1. describe this stack. The DisableRollback is true.
aws cloudformation describe-stacks --stack-name my-application 
{
    "Stacks": [
        {
            "StackId": "...",
            "StackName": "my-application",
            "ChangeSetId": "...",
            "CreationTime": "2021-09-01T03:48:10.908000+00:00",
            "LastUpdatedTime": "2021-09-01T03:53:55.837000+00:00",
            "RollbackConfiguration": {},
            "StackStatus": "CREATE_IN_PROGRESS",
            "DisableRollback": true,
            "NotificationARNs": [],
            "Capabilities": [
                "CAPABILITY_IAM"
            ],
            "Tags": [],
            "EnableTerminationProtection": false,
            "DriftInformation": {
                "StackDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}

After a moment, it starts rolling back.

However, if I deploy the stack with create-stack and --disable-rollback, it will stop in CREATE_FAILED state and not roll back.

aws cloudformation create-stack --stack-name my-application  \ 
--template-body file://template.yaml --disable-rollback --capabilities CAPABILITY_IAM

The CDK code

/// !cdk-integ pragma:ignore-assets
import * as cdk from '@aws-cdk/core';
import * as patterns from '../lib';
import * as ecs from '@aws-cdk/aws-ecs';
import * as ec2 from '@aws-cdk/aws-ec2'

class Demo extends cdk.Construct {
  constructor(scope: cdk.Construct, id: string) {
    super(scope, id);
    const vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 3, natGateways: 0 })
    const cluster = new ecs.Cluster(this, 'Cluster', {
      vpc,
      enableFargateCapacityProviders: true,
    });
    const task = new ecs.FargateTaskDefinition(this, 'Task', {
      // faulty configuration here
      cpu: 2560,
      memoryLimitMiB: 512, 
    });
    task.addContainer('nyancat', {
      image: ecs.ContainerImage.fromRegistry('public.ecr.aws/pahudnet/nyancat-docker-image:latest'),
      portMappings: [{containerPort: 80}],
      
    })
    new patterns.ApplicationLoadBalancedFargateService(this, 'Service', {
      cluster,
      circuitBreaker: { rollback: true },
      taskDefinition: task,
      taskSubnets: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
    })
  }
}

const app = new cdk.App();

const env = {
  region: process.env.CDK_DEFAULT_REGION,
  account: process.env.CDK_DEFAULT_ACCOUNT,
};

const stack = new cdk.Stack(app, 'no-rollback-demo', { env })

new Demo(stack, 'Demo')

app.synth();

What did you expect to happen?

What actually happened?

Environment

  • ** AWS CLI Version 😗* aws-cli/2.2.34 Python/3.8.8 Darwin/20.5.0 exe/x86_64 prompt/off
  • CDK CLI Version :
  • Framework Version:
  • Node.js Version:
  • OS :
  • Language (Version):

Other


This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

7reactions
mjaswanthicommented, Sep 1, 2021

Thanks Pahud. We are looking into this and will have an update shortly.

1reaction
pahudcommented, Sep 1, 2021

copy @rix0rrr

Read more comments on GitHub >

github_iconTop Results From Across the Web

AWS CDK Toolkit (cdk command) - AWS Documentation
For this reason, the CDK Toolkit lets you disable rollback by adding --no-rollback to your cdk deploy command. With this flag, failed deployments...
Read more >
Troubleshooting common AWS CDK issues
This topic describes how to troubleshoot the following issues with the AWS CDK.
Read more >
Stack failure options - AWS CloudFormation
Provide a stack name and template to the create-stack command with the disable-rollback option. The command returns the following output.
Read more >
Continue rolling back an update - AWS CloudFormation
When a stack is in the UPDATE_ROLLBACK_FAILED state, you can continue to roll it back to a working state ( UPDATE_ROLLBACK_COMPLETE ). You...
Read more >
Update CloudFormation stacks stuck in ... - Amazon AWS
In some cases, retrying the rollback doesn't resolve the error. In these cases, you must skip resources, and also acknowledge that these ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found