Make ECS tooling friendly to generating custom templates based on ECS

A frequent request about ECS is: “How can I generate Elasticsearch templates that include ECS fields and my custom fields?”

There’s many ways we could provide tooling for that: a Kibana app would probably be the most user-friendly. However the tooling in the ECS repository is very close to being able to offer this already. As a short term solution, we will be tweaking the generator in this repo to make it easier for users to manage their templates with it. As a bonus, the generator will continue generating all of the artifacts that includes people’s fields (asciidoc, csv), not only the ES templates. Some users may find that interesting.

Here’s a list of tasks that will gradually make it easier to achieve this goal:

• #494 Add the ability to include external directory of schemas
• #748 Add the ability to specify an output directory
• #746 Document how to use the ECS tooling to generate custom templates
• Let users select which artifacts they want to generate
• #737 Let users specify which ECS schemas to include, to skip the fields they will not use
• Forcefully set level: custom for any field loaded from an external directory
• Make the index template settings configurable
• Allow for adding fields within existing ECS field sets from external files

These will help address #95 and #324

[webmat]

Another note: Elastic SIEM embeds ECS and Beats field definitions already. Ask the SIEM team where the files are, perhaps there’s something you can leverage from that, elsewhere in Kibana?

[webmat]

@pmuellr Other initiatives such as the ECS logging libraries also depend on ECS artifacts and aren’t in this repo. I’ll be open to helping make this repo more amenable to consuming programmatically from other projects, such as yours or the ECS logging libraries. Let me know what challenges you encounter.

Here’s a few pointers to help you get started:

• By default, have your scripts operate on the release tags, not on the master branch. Until something gets released (and therefore tagged), it’s still open to change, or even be removed completely.
• When leveraging the ECS schema from your script, don’t try to parse what’s in schemas/*.yml, these files are for humans to read and edit. The generator does a pass on the YAML files, fills in defaults, cleans everything up, and produces two other YAML files: ecs_flat.yml and ecs_nested.yml. Same content, different layout. Both will be maintained, as the various ECS generators use them both. Flat is easier to run through for a simple file such as the csv, whereas nested keeps the details of the fieldsets & so on. Nested is a bit trickier to navigate, but more complete.

With that said, this repo cannot adopt a deployment to NPM, because by the same token, people may then request it to Rubygems, Maven, or to any other arbitrary package manager 😃 But as I said already, I’m happy to help people consume this programmatically otherwise.