How to share schemas used with fields not in ECS

Most implementations which use ECS have ECS as the basic fields but have their own fields on top. As inspiration for which fields could be added to ECS and and inspiration for other users it would be interesting if in the context of ECS people could share their used schemas for example with F5.

An current example we did with auditbeat data and the hash prefix can be found here: https://github.com/elastic/ecs/blob/a256a0cb7ca683b525f931ef030fc792848b7ae1/use-cases/auditbeat.yml#L78-L88

These fields are currently listed in use cases but commented out. A better solution is needed. One idea would be to have these use cases with the complete set of fields an user can contribute them but all fields which are not part of ECS are listed separately. The two things I worry here is that creating the fields.yml is sometimes too much overhead and sharing just a json would be easier, the other part is people might get confused on what is part of ECS and what is not.

Any ideas are more then welcome.

[djptek]

Hi @ruflin the ECS repo now has tooling which allows for users to add their own fields in conjunction with ECS

[jordansissel]

@ruflin I had another thought, that perhaps field aliases (assuming they are going to land in Elasticsearch) might be another way to solve this.

My ECS transform is done in Go right now before ingestion, for example, and I copy things as-is without modifying the values (to change types from string/date/number/etc). Instead, with field aliases, we could provide this to Elasticsearch and let it do the mappings (This assumes the field values are usable as-is without modification, though?)

For Okta, for example, I transform actor.alternateId to be ECS’ user.id. If Elasticsearch gets field aliases, then an ingest author doesn’t need to do this transform because we can tell Elasticsearch “The field user.id comes from the actor.alternateId field”. This may be the best of both worlds where we can allow search/aggs by ECS but we don’t actually modify the source data, so subject matter experts would see Okta System Logs (for example) in their native format and we can still build dashboards against the ECS schema.