[meta] Consolidate generated ES index templates

Overview

With ECS aligning versioning with the Elastic stack releases starting in 8.0, the ECS tooling could be simplified to generate a single set of Elasticsearch index template artifacts. This single generated version would align with the stack release of the same version.

For example, ECS 8.0 produces sample Elasticsearch index mappings that are compatible with Elastic Stack 8.0.

Advantages

• Simplify the ECS tooling. No additional code to generate multiple versions of the index templates to support multiple versions of ES or multiple versions of the ES index template APIs.
• Reduce the copies of index templates in the repo; simplified user experience.
• Prevent users from trying to use the curated artifacts with the wrong APIs: example
• The sample templates are maintained as examples. They should be used as a starting point for experimentation. Users are still free to generate their own variations of index templates but shouldn’t rely on ECS maintaining multiple copies.

Concerns

• Should the tooling allow users to maintain their own legacy index template artifacts (e.g. for users who maintain customer mappings)?
• The logstash-output-elasticsearch plugin depends on the ECS projects generated, legacy index template artifacts as part of their build for ECS compatibility mode. We will need to remove this dependency before we can remove the generated, legacy templates from ECS.

[yaauie]

No. We can work from the “legacy” v1 templates, and in fact doing so simplifies our codepath. We already translate the “legacy”-style index templates for ECS 1.x when vendoring them for use with Elasticaearch 8, and can use the same code-path with the stack-aligned “legacy”-style templates. Moving the Stack-aligned monolith template to V2 adds complexity because we would need to translate those back into legacy for use with ES 7.x (due to our plugins implementation).

[kgeller]

Conclusion:

Since ECS is aligning versioning with the Elastic stack releases starting in 8.0, we have simplified the ECS tooling to generate two sets of Elasticsearch index template artifacts. We will now produce 1) a composable template compatible with the v2 _index_template API and 2) a legacy index template compatible with v1 “legacy” _template API.

Previously, at every ECS version, we were generating legacy templates compatible with ES 6 and ES 7, as well as a composable template. We have now consolidated down to, for every ECS version, a single ES 8 compatible legacy template and a single composable template.