AWS Support of CloudEvents

Hi. We’re working on a bunch of event-related stuff at AWS with a view to releases in 2019. It has been internally proposed that we “Support CloudEvents” and this seems to me like a good idea. No promises, but my opinion is that if we could find a plausible way to do it, there is a high likelihood that we would.

Important This is not an official communication, this is Tim looking for options I can use internally.

Background

We already have an AWS Event envelope that is widely used across AWS chiefly through the “CloudWatch Events” service. There’s no formal schema or anything but it’s documented here.

The required top-level fields are:

• version: (Corresponds to CloudEvents specversion)
• detail-type: (type)
• source: (source)
• id: (id)
• time: (time)
• detail: (data)

Fields in AWS Events not in CloudEvents: region, account, resources.

Fields in CloudEvents not in AWS Events: datacontenttype (So far we’re all-JSON all the time.)

Problems

1. There are already huge numbers of AWS Events per second flowing through our system and there’s no API, the only documentation is of the bits on the wire. And, there are a huge number of AWS customers who are making use of this service. Therefore there is no reasonable prospect of us changing field names.
2. CloudEvents isn’t finished, but we want to ship production software in 2019. In theory, the working group could do a wholesale redesign of the attributes at any time. Can any organization plausibly claim to “Support CloudEvents” at this stage in history? It’s not obvious to me how.

Appeal for input

Before I dive any deeper, I’d welcome general comments along the lines of “No, just wait till CloudEvents is finished”, or “That is a really bad idea because X” for some value of X, or “You can effectively support CloudEvents by doing A, B, and C.” Thanks in advance.

Some notes about possibilities

1. CloudEvents could consider freezing the field names. I.e. adopt some sort of policy that whereas new attributes may be added, there’s a commitment not to change the names of the ones that exist. 1.1. And if you wanted to try making AWS an offer it couldn’t refuse, consider changing the field names to match ours. We could add support for the extra CloudEvents fields and our extras could be extensions. There are some semantic incompatibilities (e.g. our “source” field looks like a relative URI reference) but probably there are workarounds. (OK, long shot).
2. Make sure all the CloudEvents SDKs include getters (I don’t think setters?) for all the specified attributes. That way, if someone like us has CloudEvents-like constructs but with incompatible names, we could implement the SDK and then claim we’re compatible because you can always use the CloudEvents SDK to access the attributes. Is this sane?
3. At the very least we could ship libraries that transcode between AWS Events and CloudEvents. Would that be enough to make users happy?

[to-masz]

@timbray I would like to highlight what @cneijenhuis said about AWS support - it would be good to define how to use CloudEvents with AWS services like SNS/SQS.

1. whether you prefer to add support for HTTP transport binding (https://github.com/cloudevents/spec/blob/master/http-transport-binding.md#314-examples), e.g. as dedicated SNS endpoint to publish messages in that form and then to support it for HTTP endpoint delivery
2. or just write down SNS transport binding, e.g. similar to https://github.com/cloudevents/spec/blob/master/kafka-transport-binding.md#325-example

[cneijenhuis]

@moltar Tim has left AWS: https://www.tbray.org/ongoing/When/202x/2020/04/29/Leaving-Amazon