[Search] Elastic Search Engine support

What / Why

We provide easy, no/low-config support for search in Backstage using an in-memory search engine backed by Lunr. Given a sufficiently large set of Catalog entities, or a multi-pod deployment of Backstage, this approach will not be adequate.

To support these production use-cases, we’d like to add support for new elasticsearch engine.

Scope / UAT Target

• Setup
  • As an app integrator, I should be able to configure connection details to an instance of ElasticSearch in app-config.yaml and instantiate an ElasticSearchEngine in my backend using that configuration.
• Querying
  • As an app integrator, I should not have to specify how a SearchQuery (and its term, filters, types/indices, pagination) is translated into ElasticSearch’s query language.
  • As an app integrator, it should be possible to provide custom query translation logic.
• Indexing
  • As a user of Backstage Search, valid search results should return for an index both during and after the indexing process completes.
  • As an app integrator, I should not have to worry about runaway disk usage in Elastic as a result of duplicate indices and/or errors during the index process.

Implementation Notes

• Consider using a popular ElasticSearch query builder like elastic-builder in the translator.
• Consider creating new indices each time SearchEngine::index is called, using aliases to flip between “old” and “current” indices for a given type/index, then deleting the old version (e.g. creating real-[indexType]-[timestamp], pointing search-[indexType] at the most recently created version, and deleting real-[indexType]-[oldTimestamp])

[emmaindal]

@iamEAP yes, lets do a session on that end of sprint to provide more details to this issue!

[iamEAP]

Closing this out as a result of #6569! Thanks again Jussi.