[FEATURE] Migration guide from elasticsearch-py

Is your feature request related to a problem?

I’m migrating from elasticsearch-py to opensearch-py, as I’m sure many customers will be. I’d like a step by step guide to make changing over easy for me, and to make the differences clear.

My specific issue is that I use a fairly standard pattern for providing AWS credentials to Opensearch, and it does not work now I’ve changed clients. Extracting my client creation code.

from opensearchpy import OpenSearch
from requests_aws4auth import AWS4Auth
import boto3

def get_opensearch_connection() -> OpenSearch:
    region = 'us-west-2'
    service = 'es'
    credentials = boto3.Session().get_credentials()
    awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

    return OpenSearch(
        hosts=[{'host': os.environ['OpensearchEndpoint'], 'port': 443}],
        http_auth=awsauth,
        use_ssl=True,
        verify_certs=True,
    )

This results in the stack trace:

[ERROR] AttributeError: 'AWS4Auth' object has no attribute 'encode'
Traceback (most recent call last):
  File "/var/task/index.py", line 102, in handler
    es = get_opensearch_connection()
  File "/var/task/index.py", line 21, in get_opensearch_connection
    return OpenSearch(
  File "/opt/python/opensearchpy/client/__init__.py", line 188, in __init__
    self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
  File "/opt/python/opensearchpy/transport.py", line 163, in __init__
    self.set_connections(hosts)
  File "/opt/python/opensearchpy/transport.py", line 221, in set_connections
    connections = list(zip(connections, hosts))
  File "/opt/python/opensearchpy/transport.py", line 217, in _create_connection
    return self.connection_class(**kwargs)
  File "/opt/python/opensearchpy/connection/http_urllib3.py", line 150, in __init__
    self.headers.update(urllib3.make_headers(basic_auth=http_auth))
  File "/opt/python/urllib3/util/request.py", line 85, in make_headers
    headers["authorization"] = "Basic " + b64encode(b(basic_auth)).decode("utf-8")
  File "/opt/python/urllib3/packages/six.py", line 687, in b
    return s.encode("latin-1")

The pattern I’m using is the same detailed at https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-python using elasticsearch-py.

What solution would you like?

I’d like a step by step migration guide to make changing over easy for me, and to make the differences clear between the two clients.

What alternatives have you considered?

Alternatives would be for opensearch-py to have exact API and feature parity with elasticsearch-py, and for an automated migration tool to be provided.

Do you have any additional context?

No

[aetter]

Hi @SamStephens, a couple minor tweaks and I think you’ll be good to go. The big thing is importing RequestsHttpConnection and adding connection_class during client creation:

from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3

host = '' # For example, my-test-domain.us-east-1.es.amazonaws.com
port = 443
region = '' # e.g. us-west-1

service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

client = OpenSearch(
    hosts = [{'host': host, 'port': port}],
    http_compress = True, # enables gzip compression for request bodies
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)
# Create an index with non-default settings.
index_name = 'python-test-index'
index_body = {
  'settings': {
    'index': {
      'number_of_shards': 4
    }
  }
}
response = client.indices.create(index_name, body=index_body)
print('\nCreating index:')
print(response)
# Add a document to the index.
document = {
  'title': 'Moneyball',
  'director': 'Bennett Miller',
  'year': '2011'
}
id = '1'
response = client.index(
    index = index_name,
    body = document,
    id = id,
    refresh = True
)
print('\nAdding document:')
print(response)
# Search for the document.
q = 'miller'
query = {
  'size': 5,
  'query': {
    'multi_match': {
      'query': q,
      'fields': ['title^2', 'director']
    }
  }
}
response = client.search(
    body = query,
    index = index_name
)
print('\nSearch results:')
print(response)
# Delete the document.
response = client.delete(
    index = index_name,
    id = id
)
print('\nDeleting document:')
print(response)
# Delete the index.
response = client.indices.delete(
    index = index_name
)
print('\nDeleting index:')
print(response)

When we’ve done a more “formal” release of the new clients, I’ll check out the AWS documentation. Given that legacy Elasticsearch OSS remains an option on the service, there’s utility in keeping sample code for the old clients, as well.

[SamStephens]

@harshavamsi I think this issue remained open because of the discussion about https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-python using the legacy Elasticsearch OSS client, but looking at that page now it’s been updated to use opensearch-py.

The entire rest of the content of this issue is to do with me missing the RequestsHttpConnection when migrating from the legacy Elasticsearch OSS client to opensearch-py, which I think is simply my mistake, and doesn’t warrant a migration guide or anything.

Unless there’s anything further you wish to do with this issue, I’d close it.