Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CloudFormation returns Internal Failure when changing existing RDS Instance parameters

See original GitHub issue

Check #6439 for more details.

AWS Support Case ID: 6844571841

We are having issues applying any kind of changes to an existing RDS stack. We’re using aws-cdk in Typescript. AWS Support noted that Python version worked for them running on aws-cdk v1.18.0. There is a strong possibility this may be a CloudFormation issue, but we’re looking into figuring out reproduction steps.

aws-cdk team members aware of this issue: @nija-at (through #6439) @MrArnoldPalmer (through Gitter)

aws-cdk code:

    const dbInstance = new rds.DatabaseInstance(this, "VectorRds", {
      instanceIdentifier,
      engine: rds.DatabaseInstanceEngine.POSTGRES,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      engineVersion,
      instanceClass: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.SMALL),
      masterUsername: "admin",
      backupRetention: cdk.Duration.days(backupRetention),
      copyTagsToSnapshot: true,
      securityGroups: [ dbSecurityGroup ],
      storageType: rds.StorageType.IO1,
      iops: 2000,
      // enablePerformanceInsights: true, // only works on PG version 10+
      deletionProtection, // false only in development
      multiAz,
      vpc,
    });

Here is our redacted template:

{
  "Resources": {
    "VectorRdsSubnetGroup620EE5FB": {
      "Type": "AWS::RDS::DBSubnetGroup",
      "Properties": {
        "DBSubnetGroupDescription": "Subnet group for VectorRds database",
        "SubnetIds": [
          "subnet-1234567890",
          "subnet-0987654321",
          "subnet-abcdefghij"
        ]
      },
      "Metadata": {
        "aws:cdk:path": "DMVectorRdsStack/VectorRds/SubnetGroup"
      }
    },
    "VectorRdsSecret66CB2648": {
      "Type": "AWS::SecretsManager::Secret",
      "Properties": {
        "Description": {
          "Fn::Join": [
            "",
            [
              "Generated by the CDK for stack: ",
              {
                "Ref": "AWS::StackName"
              }
            ]
          ]
        },
        "GenerateSecretString": {
          "ExcludeCharacters": "\"@/\\",
          "GenerateStringKey": "password",
          "PasswordLength": 30,
          "SecretStringTemplate": "{\"username\":\"admin\"}"
        }
      },
      "Metadata": {
        "aws:cdk:path": "DMVectorRdsStack/VectorRds/Secret/Resource"
      }
    },
    "VectorRdsSecretAttachment2E39F871": {
      "Type": "AWS::SecretsManager::SecretTargetAttachment",
      "Properties": {
        "SecretId": {
          "Ref": "VectorRdsSecret66CB2648"
        },
        "TargetId": {
          "Ref": "VectorRds92A77672"
        },
        "TargetType": "AWS::RDS::DBInstance"
      },
      "Metadata": {
        "aws:cdk:path": "DMVectorRdsStack/VectorRds/Secret/Attachment/Resource"
      }
    },
    "VectorRds92A77672": {
      "Type": "AWS::RDS::DBInstance",
      "Properties": {
        "DBInstanceClass": "db.t3.small",
        "AllocatedStorage": "100",
        "BackupRetentionPeriod": 7,
        "CopyTagsToSnapshot": true,
        "DBInstanceIdentifier": "testing2",
        "DBSubnetGroupName": {
          "Ref": "VectorRdsSubnetGroup620EE5FB"
        },
        "DeletionProtection": false,
        "Engine": "postgres",
        "EngineVersion": "11.5",
        "Iops": 2000,
        "MasterUsername": {
          "Fn::Join": [
            "",
            [
              "{{resolve:secretsmanager:",
              {
                "Ref": "VectorRdsSecret66CB2648"
              },
              ":SecretString:username::}}"
            ]
          ]
        },
        "MasterUserPassword": {
          "Fn::Join": [
            "",
            [
              "{{resolve:secretsmanager:",
              {
                "Ref": "VectorRdsSecret66CB2648"
              },
              ":SecretString:password::}}"
            ]
          ]
        },
        "MultiAZ": true,
        "StorageType": "io1",
        "VPCSecurityGroups": [
          {
            "Fn::ImportValue": "dm-vector-rds-sg"
          }
        ]
      },
      "UpdateReplacePolicy": "Delete",
      "DeletionPolicy": "Delete",
      "Metadata": {
        "aws:cdk:path": "DMVectorRdsStack/VectorRds/Resource"
      }
    }
  },
  "Outputs": {
    "dmvectorrdsdnsendpoint": {
      "Description": "Vector RDS DNS endpoint",
      "Value": {
        "Fn::GetAtt": [
          "VectorRds92A77672",
          "Endpoint.Address"
        ]
      },
      "Export": {
        "Name": "dm-vector-rds-dns-endpoint"
      }
    }
  }
}

Reproduction Steps

  • Create RDS stack using CDK
  • Introduce a change (different instance type/size or multiAZ)
  • CloudFormation throws error Internal Failure

Error Log

Example of changing multiAz parameter:

Stack VectorRdsStack
 Resources
 [~] AWS::RDS::DBInstance VectorRds VectorRds92A77672 
  └─ [+] MultiAZ
      └─ true
 $ cdk deploy '*' --require-approval 'never'
 VectorRdsStack: deploying...
 VectorRdsStack: creating CloudFormation changeset...
  0/4 | 5:06:28 AM | UPDATE_IN_PROGRESS   | AWS::RDS::DBInstance                        | VectorRds (VectorRds92A77672) 
  0/4 | 5:06:28 AM | UPDATE_IN_PROGRESS   | AWS::CDK::Metadata                          | CDKMetadata 
  1/4 | 5:06:29 AM | UPDATE_FAILED        | AWS::RDS::DBInstance                        | VectorRds (VectorRds92A77672) Internal Failure
 	new DatabaseInstance (/builds/rds/dm-vector-rds/node_modules/@aws-cdk/aws-rds/lib/instance.ts:795:22)
 	\_ new VectorRdsStack (/builds/rds/dm-vector-rds/lib/vector-rds-stack.ts:37:24)
 	\_ Object.<anonymous> (/builds/rds/dm-vector-rds/bin/vector-rds.ts:11:21)
 	\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
 	\_ Module.m._compile (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:814:23)
 	\_ Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
 	\_ Object.require.extensions.<computed> [as .ts] (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:817:12)

Example of changing instanceClass parameter:

 Stack DMVectorRdsStack
 Resources
 [~] AWS::RDS::DBInstance VectorRds VectorRds92A77672 
  └─ [~] DBInstanceClass
      ├─ [-] db.t3.small
      └─ [+] db.t3.medium
 $ cdk deploy '*' --require-approval 'never'
 DMVectorRdsSgStack
 DMVectorRdsSgStack: deploying...
  ✅  DMVectorRdsSgStack (no changes)
 Outputs:
 DMVectorRdsSgStack.dmvectorrdssg = sg-02ffd1ee9287b5c53
 Stack ARN:
 arn:aws:cloudformation:us-west-2:xxxxxxxxxxx:stack/DMVectorRdsSgStack/64920690-5bf8-11ea-9a38-0a06032da4ca
 DMVectorRdsStack
 DMVectorRdsStack: deploying...
 DMVectorRdsStack: creating CloudFormation changeset...
  0/2 | 8:51:29 PM | UPDATE_IN_PROGRESS   | AWS::RDS::DBInstance                        | VectorRds (VectorRds92A77672) 
  1/2 | 8:51:29 PM | UPDATE_FAILED        | AWS::RDS::DBInstance                        | VectorRds (VectorRds92A77672) Internal Failure
 	new DatabaseInstance (/builds/rds/dm-vector-rds/node_modules/@aws-cdk/aws-rds/lib/instance.ts:795:22)
 	\_ new VectorRdsStack (/builds/rds/dm-vector-rds/lib/vector-rds-stack.ts:27:24)
 	\_ Object.<anonymous> (/builds/rds/dm-vector-rds/bin/vector-rds.ts:13:24)
 	\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
 	\_ Module.m._compile (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:814:23)
 	\_ Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
 	\_ Object.require.extensions.<computed> [as .ts] (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:817:12)
 	\_ Module.load (internal/modules/cjs/loader.js:1000:32)
 	\_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
 	\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
 	\_ main (/builds/rds/dm-vector-rds/node_modules/ts-node/src/bin.ts:226:14)
 	\_ Object.<anonymous> (/builds/rds/dm-vector-rds/node_modules/ts-node/src/bin.ts:485:3)
 	\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
 	\_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
 	\_ Module.load (internal/modules/cjs/loader.js:1000:32)
 	\_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
 	\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
 	\_ /usr/local/lib/node_modules/npm/node_modules/libnpx/index.js:268:14
  1/2 | 8:51:30 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack                  | DMVectorRdsStack The following resource(s) failed to update: [VectorRds92A77672]. 
  2/2 | 8:51:44 PM | UPDATE_COMPLETE      | AWS::RDS::DBInstance                        | VectorRds (VectorRds92A77672) 
  2/2 | 8:51:45 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                  | DMVectorRdsStack 
  3/2 | 8:51:46 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                  | DMVectorRdsStack 
  ❌  DMVectorRdsStack failed: Error: The stack named DMVectorRdsStack is in a failed state: UPDATE_ROLLBACK_COMPLETE
     at /usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:165:13
     at processTicksAndRejections (internal/process/task_queues.js:97:5)
     at waitFor (/usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:76:20)
     at Object.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:155:7)
     at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:137:24)
     at main (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:218:16)
     at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:168:9)
 The stack named DMVectorRdsStack is in a failed state: UPDATE_ROLLBACK_COMPLETE
 ERROR: Job failed: exit code 1

Environment

  • Framework Version: 1.26.0
  • Language : Typescript

Other

This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:13 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
nija-atcommented, Mar 5, 2020

If it’s alright, I’m going to go ahead and close this issue since this isn’t related to the CDK.

You may continue tracking the issue that @maiconrocha linked above for updates, or with case opened with AWS support.

Let me know if I’ve missed anything here.

0reactions
jls-tschanzccommented, Mar 6, 2020

@robertd Will do so.

I had hoped the CFN team had a way to report issues/bugs without paying for the privilege to do so; Technical Support Tickets are only available for Support Plans higher than “basic”.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Resolve the "Internal Failure" error in CloudFormation
If you're creating or updating your CloudFormation stack, you can receive an "Internal Failure" error when an operation on a resource fails.
Read more >
Top 10 Serverless Deployment Errors (and How to Fix Them)
The state returned from AWS CloudFormation is usually UPDATE_ROLLBACK_FAILED or DELETE_FAILED , and the error will look something like this:.
Read more >
Troubleshooting Amazon CloudFormation StackSets
To troubleshoot why a stack creation or update failed, open the Amazon CloudFormation console, and view the events for the stack, which will...
Read more >
CloudFormation RDS CreateInstance fails incompatible ...
That error is telling you that you have a value set in your custom parameter group that is preventing the database from starting...
Read more >
AWS CloudFormation and RDS Snapshots
In short, if you create an RDS instance from a snapshot of a “foo” database and then change the template to have a...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

S3 Bucket Policy Changes Not Recognized As A Change on CDK Deploy

Next page

cdk diff (and cdk deploy) complains about short functions in existing yaml