Better way to patch resources at the CloudFormation level to workaround gaps (escape hatch)

Currently, the only way to workaround gaps in the AWS Construct Library is to overload toCloudFormation at the Stack level and patch the synthesized output:

class MyStack extends cdk.Stack {
    // ...
    toCloudFormation() {
      const cf = super.toCloudFormation();
      cf.Resources.Pg4352TopicPg4352QueueSubscriptionDD55615C.Properties.Protocol = 'bla';
      return cf;
    }
}

Requirements

• Allow override (add/remove/modify) of the L1 properties of resources created by L2 constructs.
• Allow accessing to the strongly-typed surface area of L1 (not just raw key/values)
• Allow modifying resource metadata and other L1 resource options such as DependsOn (e.g. #777)
• Allow modifying L1 properties that do not have strong-type representation (i.e. properties that are not defined in the CFN spec yet)
• Allow adding resources that are not in the CFN spec (already supporting by directly instantiating cdk.Resource, but we should document)
• Allow changing individual values inside a complex L1 property type
• It should be possible to explicitly delete values.
• It should be possible to set a value to an empty object, array or null as those are sometimes needed by CloudFormation (e.g)

Non functional:

• It should be possible to specify overrides “close” to where the L2 is defined.
• Overrides should be applied on top of anything L2 does, such as mutations, validations, etc (which can happen at any time before synthesis, and after validation) - this means that overrides should be applied during synthesis of the resource and not before.
• Overrides should not require that the L2 will “support” them, so it will truly be an “escape hatch” for people to be able to work around missing capabilities or APIs
• Raw overrides should also be able to bypass any L1 validations, in case the CFN spec is wrong or not up-to-date
• It should be possible to assign tokens (and stringified tokens) as override values.
• Record override info in our analytics system (AWS::CDK::Metadata resource) to help prioritize L2 work and identify gaps (#785)

Design

Based on the requirements above, here’s a design proposal.

Accessing L1 resources

To get a handle on an L1 resource behind an L2, users will use the construct.findChild(id) or tryFindChild(id) and downcast it to the specific L1.

For example, to access the L1 bucket resource given an L2 bucket:

const bucketResource = bucket.findChild('Resource') as s3.cloudformation.BucketResource;

By convention, the ID of the “main” resource of L2s is Resource, so in most cases, it should be straightforward to find the child.

In more complex scenarios, users will have to consult the L2 code in order to learn the ID (for example, asg.findChild('LaunchConfig') will return autoScaling.cloudformation.LaunchConfigurationResource.

Perhaps we can improve our convention such that the IDs of the children that are not the main resource will be 1:1 with the CloudFormation resource type name, so instead of LaunchConfig we should use LaunchConfiguration.

This approach allows users to access L1 resources freely, and obtain a strongly-typed L1 type from them without requiring that the L2 layer will “expose” those resources explicitly.

findChild will fail if the child could not be located, which means that if the underlying L2 changes the IDs or structure for some reason, synthesis will fail.

It is also possible to use construct.children for more advanced queries. For example, we can look for a child that has a certain CloudFormation resource type:

const bucketResource = 
    bucket.children.find(c => (c as cdk.Resource).resourceType === 'AWS::S3::Bucket') 
    as s3.cloudformation.BucketResource;

From that point, users are interacting with the strong-typed L1 resources (which extend cdk.Resource), so we will look into how to extend their surface area to support the various requirements.

Resource Options

cdk.Resource already has a few facilities for setting resource options such as Metadata, DependsOn, etc:

const bucketResource = bucket.findChild('Resource') as s3.cloudformation.BucketResource;

bucketResource.options.metadata = { MetadataKey: 'MetadataValue' };
bucketResource.options.updatePolicy = {
    autoScalingRollingUpdate: {
        pauseTime: '390'
    }
};

bucketResource.addDependency(otherBucket.findChild('Resource') as cdk.Resource);

This will synthesize to:

{
    "Type": "AWS::S3::Bucket",
    "DependsOn": [ "Other34654A52" ],
    "UpdatePolicy": {
        "AutoScalingRollingUpdate": {
            "PauseTime": "390"
        }
    },
    "Metadata": {
        "MetadataKey": "MetadataValue"
    }
}

Raw Overrides

At the lowest level, we want to allow users to “patch” the synthesized resource definition and bypass any validations at the L1/L2 layers.

To that end, we will add a method resource.addOverride(path, value) which will allow applying arbitrary overrides to a synthesized resource. addOverride will be able to override anything under the resource definition (including resource options such as DependsOn, etc). We will also add resource.addPropertyOverride(propertyPath, value) as sugar for property overrides.

For example:

bucketResource.addOverride('Transform', 'Boom');
bucketResource.addPropertyOverride('VersioningConfiguration.Status', 'NewStatus');

Will synthesize to:

{
    "Type": "AWS::S3::Bucket",
    "Properties": {
        "VersioningConfiguration": {
            "Status": "NewStatus"
        }
    },
    "Transform": "Boom"
}

It should be possible to also assign null as an override value:

bucketResource.addPropertyOverride('Nullify.Me', null);

Synthesizes to:

{
    "Type": "AWS::S3::Bucket",
    "Properties": {
        "Nullify": {
            "Me": null
        }
    }
}

It should also possible to clear a value using an override:

bucketResource.addDeletionOverride('Delete.Me');

Overrides may freely use tokens (or stringified tokens) for values, and those will be resolved during synthesis:

bucketResource.addPropertyOverride('OtherBucketArn', otherBucket.bucketArn);

Will synthesize to:

{
    "Type": "AWS::S3::Bucket",
    "Properties": {
        "OtherBucketArn": {
            "Fn::GetAtt": [
                "Other34654A52",
                "Arn"
            ]
        }
    }
}

Strong-type property overrides

Users should also be able to define overrides for L1 resource properties via the generated property types. To that end, each generated resource will expose another property propertyOverrides of type XxxProps.

So, s3.cloudformation.BucketResource#propertyOverrides will have the type s3.cloudformation.BucketResourceProps, which allow users to define overrides for bucket resource properties as follows:

bucketResource.propertyOverrides.corsConfiguration = {
    corsRules: [
        {
            allowedMethods: [ 'GET' ],
            allowedOrigins: [ '*' ]
        }
    ]
};

Notes:

• propertyOverrides are merged into the resource on top of the values defined during initialization.
• This means that in order to delete a value, you will have to set it to null in the overrides.
• Overrides will not go through L1 validation.

Defining raw resources

It is also possible to directly use cdk.Resource to define arbitrary CloudFormation resources:

new cdk.Resource(this, 'MyResource', {
  type: 'AWS::Unknown::Resource',
  properties: {
    Foo: 'bar'
  }
});

[eladb]

Here: https://awslabs.github.io/aws-cdk/aws-construct-lib.html#access-the-aws-cloudformation-layer

[eladb]

@ccurrie-amzn see #782 regarding “turning off” validation. TL;DR, it’s non-trivial, so I am punting until we have evidence that it is actually needed/valuable.