(aws-rds): `addRotationMultiUser()` changes the username, adds `_clone` suffix

Describe the bug

After our rotator lambda ran, we discovered that the username in the secret had been change to add a _clone suffix.

❯ aws secretsmanager list-secret-version-ids --region="$AWS_REGION" --secret-id="ColdStorageWriter"
{
    "Versions": [
        {
            "VersionId": "6a00bb61-5e22-4cb4-ab46-eb6b78402d05",
            "VersionStages": [
                "AWSCURRENT",
                "AWSPENDING"
            ],
            "LastAccessedDate": "2022-06-09T17:00:00-07:00",
            "CreatedDate": "2022-06-08T15:12:32.509000-07:00",
            "KmsKeyIds": [
                "redacted"
            ]
        },
        {
            "VersionId": "ec55412e-18ee-46ce-8aa5-e5199b2ece1e",
            "VersionStages": [
                "AWSPREVIOUS"
            ],
            "LastAccessedDate": "2022-06-07T17:00:00-07:00",
            "CreatedDate": "2022-05-09T13:31:13.330000-07:00",
            "KmsKeyIds": [
                "redacted"
            ]
        }
    ],
    "ARN": "redacted",
    "Name": "ColdStorageWriter"
}

❯ aws secretsmanager get-secret-value --region="$AWS_REGION" --secret-id="ColdStorageWriter" --version-id="ec55412e-18ee-46ce-8aa5-e5199b2ece1e"
{
    "ARN": "arn:aws:secretsmanager:us-west-2:514308641592:secret:ColdStorageWriter-rZPWY6",
    "Name": "ColdStorageWriter",
    "VersionId": "ec55412e-18ee-46ce-8aa5-e5199b2ece1e",
    "SecretString": "{\"password\":\"redacted\",\"masterarn\":\"redacted did not change",\"username\":\"cold_storage_writer\",\"host\":\"redacted\",\"engine\":\"postgres\",\"proxyHost\":\"redacted\"}",
    "VersionStages": [
        "AWSPREVIOUS"
    ],
    "CreatedDate": "2022-05-09T13:31:13.330000-07:00"
}

❯ aws secretsmanager get-secret-value --region="$AWS_REGION" --secret-id="ColdStorageWriter"
{
    "ARN": "arn:aws:secretsmanager:us-west-2:514308641592:secret:ColdStorageWriter-rZPWY6",
    "Name": "ColdStorageWriter",
    "VersionId": "6a00bb61-5e22-4cb4-ab46-eb6b78402d05",
    "SecretString": "{\"password\": \"redacted", \"masterarn\": \"redacted\", \"username\": \"cold_storage_writer_clone\", \"host\": \"redacted\", \"engine\": \"postgres\", \"proxyHost\": \"redacted\"}",
    "VersionStages": [
        "AWSCURRENT",
        "AWSPENDING"
    ],
    "CreatedDate": "2022-06-08T15:12:32.509000-07:00"
}

Expected Behavior

The rotator should rotate the password without completely undocumented side-effects like, for example, changing the username.

Current Behavior

Our users with addRotationMultiUser() are getting their usernames changed.

Reproduction Steps

import { Aurora } from ‘@time-loop/cdk-aurora’; import { App, aws_ec2, aws_kms, Stack, StackProps } from ‘aws-cdk-lib’; import { Construct } from ‘constructs’; import { Namer } from ‘multi-convention-namer’;

export class AuroraDemoStack extends Stack { constructor(scope: Construct, props: StackProps) { const id = new Namer([‘aurora’, ‘demo’]); super(scope, id.pascal, props);

const vpc = aws_ec2.Vpc.fromLookup(this, 'Vpc', {
  isDefault: true,
});

const kmsKey = new aws_kms.Key(this, 'Key', {
  description: `${id.pascal} encryption key`,
});

const a = new Aurora(this, id, {
  defaultDatabaseName: 'demo',
  instances: 1, // It's just a demo
  kmsKey,
  vpc,
});

} }

// for development, use account/region from cdk cli const devEnv = { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION, };

const app = new App();

new AuroraDemoStack(app, { env: devEnv });

app.synth();

Possible Solution

I think that the decision to SAM in the rotator functions has been a mess. These rotator functions should be rewritten in TypeScript and inlined into the code and actually managed.

Related issues

• https://github.com/aws/aws-cdk/issues/18249 - no ownership on the aws-cdk side for keeping the SAM rotators up to date
• https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/issues/84 - no ownership on the SAM side for… well… anything.

Additional Information/Context

No response

CDK CLI Version

2.27.0 (build 8e89048)

Framework Version

2.27.0

Node.js Version

v16.13.1

OS

Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101 arm64

Language

Typescript

Language Version

4.7.3

Other information

No response

[jogold]

Also, I recognize that set of excludeCharacters, it’s the same ones that were breaking DMS a couple of years back. Not happy to see that stuff return.

Looks like they added ExcludeCharacters so in the mean time you should be able to do:

const rotationSchedule = mySecret.addRotationSchedule('Rotation', {
  hostedRotation: HostedRotation.mysqlSingleUser({ vpc: myVpc }),
});
const cfnRotationSchedule = rotationSchedule.node.defaultChild as CfnRotationSchedule;
cfnRotationSchedule.addPropertyOverride('HostedRotationLambda.ExcludeCharacters', '<your chars>');

Will open a PR to add this option.

[jogold]

WRT the excludeCharacters stuff, does the new stuff default to DMS compatible secrets?

The code is at https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/80b407354909519cf4f2d744c2d9dace09b05d39/SecretsManagerRDSPostgreSQLRotationSingleUser/lambda_function.py#L118 => ':/@"\'\\'

• create a SecretsManager secret with the necessary fields (I think we can do this in cdk directly)

Yes, you can use a secretsmanager.Secret or a rds.DatabaseSecret, do not forgot to attach() it to your instance/cluster (this adds the DB info in the secret)

• write a CustomResource (https://github.com/time-loop/cdk-aurora/blob/main/src/aurora.provisioner.ts) that uses the admin user to connect to the RDS, create the user / set the password

Yes or create it by any other means

• add a HostedRotation

You can add it immediately, it will only start/succeed when the user is correctly created in the DB