(rds): Is it really that easy to destroy a database?

The Question

This is the situation: we are developing an application which has CDK stack using the RDS module for Postgresql. The application is normally deployed using Github Actions and the database has (for now) only ONE user defined.

One day one developer deployed from their local environment (we should not do this, but it happened) and all of the sudden, the database in RDS disappeared.

During the deployment, CDK detected a change: that only change is in the credentials of the database; the value in the live environment differs (of course) from the value from the developer’s laptop. This single change, for some reason, made CDK (or Cloudformation) take the decision to destroy the database and recreating it from scratch. This is part of the Cloudformation logs:

UPDATE_IN_PROGRESS: Requested update requires the creation of a new physical resource; hence creating one.

I know that there are options to avoid this potential disaster, but they should be enabled by default.

Our running theory is that CDK is not able to understand if/how to change credentials in a running db, hence it goes with the simplest solution: destroy/create. We tried more than once, and this is apparently the effect.

This is the relevant part of our ifrastructure:

    const instance = new rds.DatabaseInstance(this, 'SomeDatabaseInstance', {
      vpc,
      engine: rds.DatabaseInstanceEngine.postgres({
        version: rds.PostgresEngineVersion.VER_12_3,
      }),
      vpcPlacement: {
        subnetType: SubnetType.PUBLIC,
      },
      databaseName: config.database.name,
      credentials: {
        username: config.database.username,
        password: secret,
      },
      instanceType: config.database.instanceType,
    });

Environment

• CDK CLI Version: 1.73.0
• Module Version: “@aws-cdk/aws-rds@1.73.0”
• Node.js Version: 14.13.0
• OS: macOS Catalina
• Language (Version): TypeScript (4.0.3)

[skinny85]

Of course. Changing the username of the master user requires the replacement of the database. But I don’t see why you would ever need to change that value (including in the developer stacks).

[skinny85]

I’m going to resolve this one. @claudioc please comment if you have any more questions!