(core): Add source code relative file path to the CloudFormation template Metadata

Description

Let’s take usermanagement-backend AWS CDK application as an example. It contains a pipeline that defines a production stage. The production stage contains a stateful stack with a database based on a DynamoDB table. Below is the metadata for the DynamoDB table synthesized for the above part of the application:

{
  "Resources": {
    "DatabaseTableF104A135": {
      "Type": "AWS::DynamoDB::Table",
      <snip>,
      "Metadata": {
        "aws:cdk:path": "UserManagementBackend-Pipeline/UserManagementBackend-Prod/Stateful/Database/Table/Resource"
      }
    },
  },
<snip>
}

I would like to have the AWS CDK source code relative file path and line number included in the CloudFormation template Metadata section using a separate key (e.g. aws:cdk:code). Here:

{
  "Resources": {
    "DatabaseTableF104A135": {
      "Type": "AWS::DynamoDB::Table",
      <snip>,
      "Metadata": {
        "aws:cdk:code": "database/infrastructure.py#L18"
        "aws:cdk:path": "UserManagementBackend-Pipeline/UserManagementBackend-Prod/Stateful/Database/Table/Resource"
      }
    },
  },
<snip>
}

Use Case

When I run cfn-nag/cfn-guard on a synthesized CloudFormation template, I need to find the AWS CDK source code that defines the related resource. It is difficult to find that source code based on the aws:cdk:path metadata, because I need to follow construct IDs. This can be even more difficult if I use constants for parts of the construct ID name (e.g., application name as prefix for the stack). Having the source code file path and line number will save time on understanding and fixing issues reported by the aforementioned tools.

Proposed Solution

AWS CDK to add the source file path and line number to the CloudFormation template Metadata section during synthesis.

Other information

No response

Acknowledge

• I may be able to implement this feature request
• This feature might incur a breaking change

[RomainMuller]

Just had a chat w/ @alexpulver about this, and on the jsii side of things, it boils down to implementing https://github.com/aws/jsii/issues/47, or at least a sub-set of it:

• We could start off by only taking care of the “into node” side of the problem: assuming the majority use-case here is trying to relate CfnGuard offenses to the user’s “own” code, the interesting call paths will mostly originate from outside of node (e.g: Python -> Node).
  • When arriving in node, the jsii kernel library might temporarily replace the implementation of Error.captureStackTrace to insert “artificial” frames received from the jsii API request object.
  • We might want to expose Error.stackTraceLimit across the language boundary to limit stack trace sizes to the user’s desired frame count.
  • The callback flow (“from node”) is also interesting in the broader feature’s context, but might be trickier to fully implement.
• This will require collecting stack traces on the caller site, possibly frequently, which can be expensive (both in time and memory), so this feature might need to be opt-in (via environment variable, for example)

Jsii support for cross-language stack traces doesn’t necessarily entirely solve the problem at hand (but is probably necessary to achieve solutions):

• Some constructs are mutable, and the origin of a CfnLint offense might not be at the construct’s instantiation site.
• Lazy instance providers are evaluated during synthesis, often in a context that does not originate from outside of node. It might be necessary to capture the stack trace at Lazy creation time to provide all necessary context.

---

In terms of next steps… I reckon we could start by prototyping this for Python (that work has to be prioritized before it can be started), and use this to validate whether this is sufficient to address the problem – in particular, it’ll be necessary to validate the assumption that starting with only “into node” stack trace continuity is sufficient for the majority use-case.

[alexpulver]

Commenting to keep this issue open