Support AWS OpenSearch / Elasticsearch

Is your feature request related to a problem? Please describe. As of a couple months ago, the Elasticsearch organization has made the official python elasticsearch plugin incompatible with Amazon supported OpenSearch. If you fire up Superset using the current helm chart and attempt to connect to a recently deployed AWS “Elasticsearch” - which is now an Apache 2.0 licensed OpenSearch - you will receive the ambiguous error from Superset

superset 2021-09-16 17:34:09,348:WARNING:superset.views.base:[SupersetError(message='(builtins.NoneType) None\n(Background on this error at: http://sqlalche.me/e/13/dbapi)', error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: 'GENERIC_DB_ENGINE_ERROR'>, level=<ErrorLevel.ERROR: 'error'>, extra={'engine_name': 'ElasticSearch (SQL API)', 'issue_codes': [{'code': 1002, ' 
 message': 'Issue 1002 - The database returned an unexpected error.'}]})]                                                                                                                                                                                                                                                                                                              

If I run python directly on the Superset pod and connect to AWS “OpenSearch/Elasticsearch” cluster, i get the more detailed error from the elasticsearch python module directly:

elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch

It would be nice if Superset could support OpenSearch natively.

Describe the solution you’d like

• At the least, highlight in the docs that the elasticsearch plugin needs downgrading to be compatible with AWS Elasticsearch/OpenSearch clusters for the time being.
• Find a way to support OpenSearch natively. Quite annoyingly, I think AWS only published a java library for OpenSearch. An official python library that could be used by Sqlalchemy is needed I believe.

Describe alternatives you’ve considered I am currently using the following boostrapScript which downgrades the elasticsearch python plugin to the last known compatible version (before Elasticsearch became hostile to AWS, rightfully or not).

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip uninstall elasticsearch && \
  pip install \
    elasticsearch==7.13.4 \
    elasticsearch-dbapi==0.2.5 \
    elasticsearch-dbapi[opendistro] \
    psycopg2==2.8.5 \
    redis==3.2.1 && 
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

Additional context Ultimately, the fix may need to be made in SQLALCHEMY since that is the middle man between Superset and the databases, but I thought it was worth requesting here in case anyone else is trying to connect to an Amazon “Elasticsearch”/Opensearch cluster and was having difficulty.

[dpgaspar]

Hi @stockholmux, I’m a maintainer/creator of elasticsearch-dbapi. I’ll look into including opensearch-py on elasticsearch-dbapi

Thank you!

[stockholmux]

👋 I’m on the OpenSearch team.

Here is some background on the client issue that appears to be causing this. I’m only vaguely familiar with SuperSet but I believe the issue originates here:

https://github.com/preset-io/elasticsearch-dbapi/blob/master/setup.py#L31

Effectively, Elasticsearch clients past will only work with non-OSS Elasticsearch moving forward (this break also affects Elasticsearch versions prior to 7.12 IIRC). The break occurred at ES Client v7.14, so the way by including elasticsearch>7 new installs are pulling the latest which includes the blocking code.

Opensearch clients (python client here) will work with both OpenSearch and ES and are syntax compatible. Not sure how SuperSet as a project wants to handle this, but I’m glad to help personally and get some other folks from our side to contribute.