Event keys

Event streams often have an ordering which is important for a set of events for the same entity, but not for events of different entities. For example, consider events for a user management system, you might have user created and user deleted events. Here’s an example stream:

eventID: 1, eventType: com.example.user.created, data: alice
eventID: 2, eventType: com.example.user.created, data: bob
eventID: 3, eventType: com.example.user.deleted, data: bob

Now in this case, it matters which order events 2 and 3 are handled in, since they are both for the user called bob, user bob cannot be deleted before he was created, logically that doesn’t make sense. However, event 1 and 2 can be handled in any order, since they are for two users, alice and bob, it doesn’t matter which order they are handled.

For performance and scalability reasons, many message transports, such as Kafka and Kinesis, implement partitioning of event streams. This partitioning however can cause events to be processed out of order. This could happen if two events that need to be processed in order are published to two different partitions. To ensure that message ordering is maintained, these transports typically implement a concept of message keys, where each message has a key associated with it that correlates to the entity that the event that the message carries is for, and all messages with the same key go to the same partition.

This key could be expressed in the CloudEvents spec using an optional eventKey field of type Object.

A use case where this might be helpful is if there was some middleware doing a generic transformation of events from one source to another, as it would allow the transformation to be done while maintaining the event ordering, without any knowledge of the domain data or underlying transports. The middleware would only need to ensure that the eventKey from incoming events was propagated to outgoing events.

Assigned to: @jroper

[jroper]

#191 deals with ordering/sequence numbers.

[jroper]

Copying the source attribute from the spec for context:

source

• Type: URI
• Description: This describes the event producer. Often this will include information such as the type of the event source, the organization publishing the event, and some unique idenfitiers. The exact syntax and semantics behind the data encoded in the URI is event producer defined.
• Constraints:
  • REQUIRED

I think the event key definitely can (and, based on the wording above, perhaps should) be encoded in the source, though the source would contain more than just the event key, since it contains the organization publishing the event, and perhaps other data such as maybe a unique key per event. The problem with using the source to encode the event key though is that I don’t think it would achieve the goal of being able to do producer agnostic transformations while maintaining ordering, since the semantics are producer defined, and so there would be no way to know how to extract the event key from the source.