Propose new top-level prefix 'related'

What I really like about ECS is the well-defined semantics of the data model. I understand we’re still evolving what that means, of course. One downside to the way that Kibana is structured to analyzed event-based, time-based data, is that it’s impossible to pivot across fields that may have the same content, but different field names. Of course, this is one item that ECS aims to unify, but semantics have to come first.

I’d like to propose a new top-level object called related. This object will hold fields that are lists of necessary types, which allows a single-click pivot to related records where the data may exist in different fields. Some examples:

• related.ip: List of related IPs (IPv4 or IPv6)
• related.hostname: List of related DNS hostname
• related.id: List of related event IDs
• related.hash: List of all hashes listed in the event

We do this today in RockNSM, but under the field names like @meta.related_id, @meta.related_ip, etc. I’m in the process of migrating this over to ECS and I think it would be especially useful to have this across other datasets.

A tangible example: Bro files log.

The bro files log has a unique identifier for a given analyzed file. Usually, a file was analyzed as part of one or more network data streams. An extreme, but not uncommon example, is a file transferred over the bittorrent protocol. In this case, bro tracks a single file that was transferred from and to many, many hosts. In this case, the following transformation captures the relevant pivotable data:

• fuid: copy to event.id and related.id
• tx_hosts: copy to related.ip
• rx_hosts: copy to related.ip
• conn_uids: copy to related.id
• md5: copy to related.hash
• sha1: copy to related.hash
• sha256: copy to related.hash
• sha512: copy to related.hash

Of course, data needs to be moved around to make the rest of it conformant, but now we can pivot to the related connection-oriented logs and events from other data sources, that perhaps match hashes.

[praseodym]

Should the ECS index template also include copy_to definitions to create the related fields, e.g. add copy_to: "related.ip" to the source.ip template, to ensure that the related fields are always filled as expected?

[ebeahan]

The last outstanding item from the original proposal is establishing something equivalent to related.id in ECS.

We created meta-issue ##1547 to assess correlated event support in ECS. Closing in favor of the meta issue.