(rds): secret rotation application times out before rotation completes

What is the problem?

The secret rotation application times out before rotation completes and rotation fails.

Reproduction Steps

// create DB with generated master DB secret
const db = new rds.DatabaseInstance(this, 'Database', {
  engine: rds.DatabaseInstanceEngine.postgres({
    version: rds.PostgresEngineVersion.VER_13_4,
  }),
  instanceType: ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE4_GRAVITON, ec2.InstanceSize.MICRO),
  vpc,
});
// add rotation to DB master secret
db.addRotationSingleUser({
  automaticallyAfter: cdk.Duration.days(30),
});

// create secondary DB secret (used by apps to connect to DB)
const dbUserSecret = new rds.DatabaseSecret(this, 'DbUserSecret', {
  username: 'app_user',
  masterSecret: dbAdminUserSecret,
});
// attach secondary DB secret
const dbUserSecretAttached = dbUserSecret.attach(db);
// add rotation to secondary DB secret
new secretsmanager.SecretRotation(this, 'DatabaseUserSecretRotation', {
  application: secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER,
  secret: db.secret!,
  target: db,
  vpc,
  automaticallyAfter: cdk.Duration.days(30),
});

I log into the DB using the master user postgres and the generated password and create the secondary user app_user with password secret_passwd. I’m now able to log into the DB using the user app_user with password secret_passwd.

Next, I trigger secret rotation for the secondary DB secret:

aws secretsmanager rotate-secret --secret-id <SECONDARY_DB_SECRET_ID>

Alternatively: under AWS Console > AWS Secrets Manager > Secrets > DbUserSecretXXXXXXXX-yyyyyyyyyyyy, press Rotate secret immediately

What did you expect to happen?

After triggering secret rotation, I can log into the DB using the newly generated password and the old password no longer works.

What actually happened?

Secret rotation is triggered successfully, however, I’m not able to login with the newly generated password. The old password secret_passwd still works.

Looking at the CloudWatch Logs of the secret rotation application, I can see that the function repeatedly times out:

2021-11-01T17:29:04.505+01:00	START RequestId: 428afaa9-e114-43b2-8054-b71239dfb8b5 Version: $LATEST
2021-11-01T17:29:04.770+01:00	[INFO] 2021-11-01T16:29:04.769Z 428afaa9-e114-43b2-8054-b71239dfb8b5 Found credentials in environment variables.
2021-11-01T17:34:04.611+01:00	2021-11-01T16:34:04.611Z 428afaa9-e114-43b2-8054-b71239dfb8b5 Task timed out after 300.10 seconds
2021-11-01T17:34:04.611+01:00	END RequestId: 428afaa9-e114-43b2-8054-b71239dfb8b5
2021-11-01T17:34:04.611+01:00	REPORT RequestId: 428afaa9-e114-43b2-8054-b71239dfb8b5 Duration: 300100.73 ms Billed Duration: 300000 ms Memory Size: 128 MB Max Memory Used: 69 MB Init Duration: 364.56 ms
2021-11-01T17:35:45.040+01:00	START RequestId: 32583674-0f79-4c74-8971-07f594f8bb65 Version: $LATEST
2021-11-01T17:35:45.274+01:00	[INFO] 2021-11-01T16:35:45.274Z 32583674-0f79-4c74-8971-07f594f8bb65 Found credentials in environment variables.
2021-11-01T17:40:45.147+01:00	2021-11-01T16:40:45.147Z 32583674-0f79-4c74-8971-07f594f8bb65 Task timed out after 300.10 seconds
2021-11-01T17:40:45.147+01:00	END RequestId: 32583674-0f79-4c74-8971-07f594f8bb65
2021-11-01T17:40:45.147+01:00	REPORT RequestId: 32583674-0f79-4c74-8971-07f594f8bb65 Duration: 300101.37 ms Billed Duration: 300000 ms Memory Size: 128 MB Max Memory Used: 31 MB

Triggering rotation again gives the following error:

An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn't complete. That rotation will be reattempted.

CDK CLI Version

1.130.0

Framework Version

1.130.0

Node.js Version

v14.17.6

OS

macOS 12.0.1

Language

Typescript

Language Version

4.4.4

Other information

No response

[arockett]

There’s a bug in VPC selection for addRotationMultiUser.

Even if you pass in props for the subnet type to place the rotation lambda, it will always get placed in the same subnets as the cluster. See https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-rds/lib/cluster.ts line 611 for the addRotationMultiUser function. Notice how the ‘…options’ line gets overridden by the cluster subnets.

The addRotationSingleUser function doesn’t have this problem because the options are injected after the defaults. See the same file as above.

Created new issue to track: https://github.com/aws/aws-cdk/issues/19233

[jogold]

From my findings, the rotation Lambda is put in the same subnet than the DB. Secret rotation only works if the target DB is located in a private subnet (ec2.SubnetType.PRIVATE_WITH_NAT). It fails if the DB is located in an isolated subnet (ec2.SubnetType.PRIVATE_ISOLATED) or a public subnet (ec2.SubnetType.PUBLIC).

@asterikx Actually you can deploy the rotation Lambda anywhere if you use a VPC endpoint for Secrets Manager.

To close this issue I suggest the following:

• Expose vpcSubnets in addRotationSingleUser() and addRotationMultiUser()
• Improve doc to clearly explain that the rotation Lambda needs either internet connectivity or a VPC endpoint for Secrets Manager

@skinny85 wdyt? I can work on the PR for this.