Support for CloudWatch InsightRule as a Higher Level Construct

In 2019, CloudWatch announced a new product called “Contributor Insights”. It allows you to see high cardinality Top-N data of things like:

• Highest rate of throttle keys in DDB
• Highest callers in APIGateway
• Highest error rate of callers

or anything that you configure from your logs. It’s truly amazing. I see that it exists as a Cloudformation resource and an autogenerated resource

Insights break a lot of the previous paradigms though:

• you can use new statistics types to slice out data within an insight rule. As an example, Top-100 shows the top 100 accessed keys in DDB. For every datapoint, you have 2 new statistics of “UniqueContributors” and “MaxContributorValue”, this means for DDB you could do a metric graph of “percent keys throttled” where you take INSIGHT_RULE_METRIC(DDB-ThrottledKeys, UniqueContributors)/INSIGHT_RULE_METRIC(DDB-AccessedKeys, UniqueContributors) which is pretty cool. Those statistics are only for Insights today though…
• It’s properties are weird in a dashboard compared to that of a metrics object (e.g. no “metrics” property).

Use Case

I use Contributor Insights and want to put it in dashboards at a higher level than the autogenerated resource 😃

Proposed Solution

I need to take a look a bit more and I kinda want to use this issue as the place for that discussion.

There are a lot of things that don’t jive well if I were to make this a Metric…or even an IMetric. As an example, the current implementation of statistic works well with the superset of both {Metric, Insight} but if I were to add the 2 new statistics mentioned above, you would have 2 statistics that only are valid on an Insight, which I assume most people don’t use today. That would be…odd. Though technically it is a statistic… ¯\(ツ)/¯

This goes down the rabbit hole. Like metrics have namespaces, dimensions, all this stuff that insights do not have. It makes me wonder “is this really a metric”. Answer is probably not, so we would need to create something else that would work well with Dashboards/Alarms that’s “similar to a metric but not a metric”.

Other

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[DRNagar]

Is there an update on this?

[NetaNir]

Hi @iph!

I agree that we shouldn’t force this into a Metric and instead create a new InsightRule type. As for the implementation, I suggest we do it in phases, lets start with implementing the building blocks, then work backward from some rules and use cases.

For the first version lets have RuleBody and RuleName (make both required, we can relax it laster). The interesting one is the RuleBody, I agree that we should plan for another scheme release, but I don’t see a need to create a Scheme type, we can implement different RuleBody for each scheme, i.e RuleBodyV1, this has the advantage of being more explicit.

Lets start with structuring RuleBody , from the CW docs, see comments inline:

    "LogGroupNames": [
        "API-Gateway-Access-Logs*",
        "Log-group-name2"
    ],
    "LogFormat": "JSON", // Enum
    "Contribution": { 
        "Keys": [
            "$.ip"
        ],
        "ValueOf": "$.requestBytes",
        "Filters": [ // lets make it a type
            {
                "Match": "$.httpMethod",
                "In": [
                    "PUT"
                ]
            }
        ]
    },
    "AggregateOn": "Sum" // Enum

Lets create an Interface for IRuleBodyV1 with a render(). In the next version we can create convenient methods like RuleBodyV1.FromLogGroup() .

WDYTH?