CloudFormation returns Internal Failure when changing existing RDS Instance parameters
See original GitHub issueCheck #6439 for more details.
AWS Support Case ID: 6844571841
We are having issues applying any kind of changes to an existing RDS stack. We’re using aws-cdk in Typescript. AWS Support noted that Python version worked for them running on aws-cdk v1.18.0. There is a strong possibility this may be a CloudFormation issue, but we’re looking into figuring out reproduction steps.
aws-cdk team members aware of this issue: @nija-at (through #6439) @MrArnoldPalmer (through Gitter)
aws-cdk code:
const dbInstance = new rds.DatabaseInstance(this, "VectorRds", {
instanceIdentifier,
engine: rds.DatabaseInstanceEngine.POSTGRES,
removalPolicy: cdk.RemovalPolicy.DESTROY,
engineVersion,
instanceClass: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.SMALL),
masterUsername: "admin",
backupRetention: cdk.Duration.days(backupRetention),
copyTagsToSnapshot: true,
securityGroups: [ dbSecurityGroup ],
storageType: rds.StorageType.IO1,
iops: 2000,
// enablePerformanceInsights: true, // only works on PG version 10+
deletionProtection, // false only in development
multiAz,
vpc,
});
Here is our redacted template:
{
"Resources": {
"VectorRdsSubnetGroup620EE5FB": {
"Type": "AWS::RDS::DBSubnetGroup",
"Properties": {
"DBSubnetGroupDescription": "Subnet group for VectorRds database",
"SubnetIds": [
"subnet-1234567890",
"subnet-0987654321",
"subnet-abcdefghij"
]
},
"Metadata": {
"aws:cdk:path": "DMVectorRdsStack/VectorRds/SubnetGroup"
}
},
"VectorRdsSecret66CB2648": {
"Type": "AWS::SecretsManager::Secret",
"Properties": {
"Description": {
"Fn::Join": [
"",
[
"Generated by the CDK for stack: ",
{
"Ref": "AWS::StackName"
}
]
]
},
"GenerateSecretString": {
"ExcludeCharacters": "\"@/\\",
"GenerateStringKey": "password",
"PasswordLength": 30,
"SecretStringTemplate": "{\"username\":\"admin\"}"
}
},
"Metadata": {
"aws:cdk:path": "DMVectorRdsStack/VectorRds/Secret/Resource"
}
},
"VectorRdsSecretAttachment2E39F871": {
"Type": "AWS::SecretsManager::SecretTargetAttachment",
"Properties": {
"SecretId": {
"Ref": "VectorRdsSecret66CB2648"
},
"TargetId": {
"Ref": "VectorRds92A77672"
},
"TargetType": "AWS::RDS::DBInstance"
},
"Metadata": {
"aws:cdk:path": "DMVectorRdsStack/VectorRds/Secret/Attachment/Resource"
}
},
"VectorRds92A77672": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBInstanceClass": "db.t3.small",
"AllocatedStorage": "100",
"BackupRetentionPeriod": 7,
"CopyTagsToSnapshot": true,
"DBInstanceIdentifier": "testing2",
"DBSubnetGroupName": {
"Ref": "VectorRdsSubnetGroup620EE5FB"
},
"DeletionProtection": false,
"Engine": "postgres",
"EngineVersion": "11.5",
"Iops": 2000,
"MasterUsername": {
"Fn::Join": [
"",
[
"{{resolve:secretsmanager:",
{
"Ref": "VectorRdsSecret66CB2648"
},
":SecretString:username::}}"
]
]
},
"MasterUserPassword": {
"Fn::Join": [
"",
[
"{{resolve:secretsmanager:",
{
"Ref": "VectorRdsSecret66CB2648"
},
":SecretString:password::}}"
]
]
},
"MultiAZ": true,
"StorageType": "io1",
"VPCSecurityGroups": [
{
"Fn::ImportValue": "dm-vector-rds-sg"
}
]
},
"UpdateReplacePolicy": "Delete",
"DeletionPolicy": "Delete",
"Metadata": {
"aws:cdk:path": "DMVectorRdsStack/VectorRds/Resource"
}
}
},
"Outputs": {
"dmvectorrdsdnsendpoint": {
"Description": "Vector RDS DNS endpoint",
"Value": {
"Fn::GetAtt": [
"VectorRds92A77672",
"Endpoint.Address"
]
},
"Export": {
"Name": "dm-vector-rds-dns-endpoint"
}
}
}
}
Reproduction Steps
- Create RDS stack using CDK
- Introduce a change (different instance type/size or multiAZ)
- CloudFormation throws error
Internal Failure
Error Log
Example of changing multiAz
parameter:
Stack VectorRdsStack
Resources
[~] AWS::RDS::DBInstance VectorRds VectorRds92A77672
└─ [+] MultiAZ
└─ true
$ cdk deploy '*' --require-approval 'never'
VectorRdsStack: deploying...
VectorRdsStack: creating CloudFormation changeset...
0/4 | 5:06:28 AM | UPDATE_IN_PROGRESS | AWS::RDS::DBInstance | VectorRds (VectorRds92A77672)
0/4 | 5:06:28 AM | UPDATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata
1/4 | 5:06:29 AM | UPDATE_FAILED | AWS::RDS::DBInstance | VectorRds (VectorRds92A77672) Internal Failure
new DatabaseInstance (/builds/rds/dm-vector-rds/node_modules/@aws-cdk/aws-rds/lib/instance.ts:795:22)
\_ new VectorRdsStack (/builds/rds/dm-vector-rds/lib/vector-rds-stack.ts:37:24)
\_ Object.<anonymous> (/builds/rds/dm-vector-rds/bin/vector-rds.ts:11:21)
\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
\_ Module.m._compile (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:814:23)
\_ Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
\_ Object.require.extensions.<computed> [as .ts] (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:817:12)
Example of changing instanceClass
parameter:
Stack DMVectorRdsStack
Resources
[~] AWS::RDS::DBInstance VectorRds VectorRds92A77672
└─ [~] DBInstanceClass
├─ [-] db.t3.small
└─ [+] db.t3.medium
$ cdk deploy '*' --require-approval 'never'
DMVectorRdsSgStack
DMVectorRdsSgStack: deploying...
✅ DMVectorRdsSgStack (no changes)
Outputs:
DMVectorRdsSgStack.dmvectorrdssg = sg-02ffd1ee9287b5c53
Stack ARN:
arn:aws:cloudformation:us-west-2:xxxxxxxxxxx:stack/DMVectorRdsSgStack/64920690-5bf8-11ea-9a38-0a06032da4ca
DMVectorRdsStack
DMVectorRdsStack: deploying...
DMVectorRdsStack: creating CloudFormation changeset...
0/2 | 8:51:29 PM | UPDATE_IN_PROGRESS | AWS::RDS::DBInstance | VectorRds (VectorRds92A77672)
1/2 | 8:51:29 PM | UPDATE_FAILED | AWS::RDS::DBInstance | VectorRds (VectorRds92A77672) Internal Failure
new DatabaseInstance (/builds/rds/dm-vector-rds/node_modules/@aws-cdk/aws-rds/lib/instance.ts:795:22)
\_ new VectorRdsStack (/builds/rds/dm-vector-rds/lib/vector-rds-stack.ts:27:24)
\_ Object.<anonymous> (/builds/rds/dm-vector-rds/bin/vector-rds.ts:13:24)
\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
\_ Module.m._compile (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:814:23)
\_ Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
\_ Object.require.extensions.<computed> [as .ts] (/builds/rds/dm-vector-rds/node_modules/ts-node/src/index.ts:817:12)
\_ Module.load (internal/modules/cjs/loader.js:1000:32)
\_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
\_ main (/builds/rds/dm-vector-rds/node_modules/ts-node/src/bin.ts:226:14)
\_ Object.<anonymous> (/builds/rds/dm-vector-rds/node_modules/ts-node/src/bin.ts:485:3)
\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
\_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
\_ Module.load (internal/modules/cjs/loader.js:1000:32)
\_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
\_ /usr/local/lib/node_modules/npm/node_modules/libnpx/index.js:268:14
1/2 | 8:51:30 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack | DMVectorRdsStack The following resource(s) failed to update: [VectorRds92A77672].
2/2 | 8:51:44 PM | UPDATE_COMPLETE | AWS::RDS::DBInstance | VectorRds (VectorRds92A77672)
2/2 | 8:51:45 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | DMVectorRdsStack
3/2 | 8:51:46 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | DMVectorRdsStack
❌ DMVectorRdsStack failed: Error: The stack named DMVectorRdsStack is in a failed state: UPDATE_ROLLBACK_COMPLETE
at /usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:165:13
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at waitFor (/usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:76:20)
at Object.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:155:7)
at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:137:24)
at main (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:218:16)
at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:168:9)
The stack named DMVectorRdsStack is in a failed state: UPDATE_ROLLBACK_COMPLETE
ERROR: Job failed: exit code 1
Environment
- Framework Version: 1.26.0
- Language : Typescript
Other
This is 🐛 Bug Report
Issue Analytics
- State:
- Created 4 years ago
- Comments:13 (9 by maintainers)
Top Results From Across the Web
Resolve the "Internal Failure" error in CloudFormation
If you're creating or updating your CloudFormation stack, you can receive an "Internal Failure" error when an operation on a resource fails.
Read more >Top 10 Serverless Deployment Errors (and How to Fix Them)
The state returned from AWS CloudFormation is usually UPDATE_ROLLBACK_FAILED or DELETE_FAILED , and the error will look something like this:.
Read more >Troubleshooting Amazon CloudFormation StackSets
To troubleshoot why a stack creation or update failed, open the Amazon CloudFormation console, and view the events for the stack, which will...
Read more >CloudFormation RDS CreateInstance fails incompatible ...
That error is telling you that you have a value set in your custom parameter group that is preventing the database from starting...
Read more >AWS CloudFormation and RDS Snapshots
In short, if you create an RDS instance from a snapshot of a “foo” database and then change the template to have a...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
If it’s alright, I’m going to go ahead and close this issue since this isn’t related to the CDK.
You may continue tracking the issue that @maiconrocha linked above for updates, or with case opened with AWS support.
Let me know if I’ve missed anything here.
@robertd Will do so.
I had hoped the CFN team had a way to report issues/bugs without paying for the privilege to do so; Technical Support Tickets are only available for Support Plans higher than “basic”.