Getting a JSONDecodeError 'Unterminated string starting at: ...' when hitting an AWS API Gateway that proxies to an elx

Elasticsearch version (bin/elasticsearch --version): 7.8.1

elasticsearch-py version (elasticsearch.__versionstr__): 7.8.1

Please make sure the major version matches the Elasticsearch server you are running.

Description of the problem including expected versus actual behavior: Hitting search with a specific query returns a JSONDecodeError shown in the title when it looks like it’s trying to deserialize the response. The stack trace is

Traceback (most recent call last):
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/elasticsearch/serializer.py", line 97, in loads
    return json.loads(s)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Unterminated string starting at: line 1 column 7823 (char 7822)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 458, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 103, in wrapper
    return fn(*args, **kwargs)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/resources/marketplace.py", line 27, in get
    res = view_mode_to_fn[view_mode](**request_args)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/resources/marketplace.py", line 46, in get_default_view_data
    categories = marketplace_es.get_categories(user.preference.materials_vendor_ids)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/elastic/marketplace_search.py", line 49, in get_categories
    raw_result = self.app.elasticsearch.search(index=self.INDEX_NAME, body=categories_query)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 139, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 1479, in search
    return self.transport.perform_request(
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 384, in perform_request
    data = self.deserializer.loads(
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/elasticsearch/serializer.py", line 143, in loads
    return deserializer.loads(s)
  File "/home/luke/Projects/takeoffs/takeoffs-app/server/venv/lib/python3.8/site-packages/elasticsearch/serializer.py", line 99, in loads
    raise SerializationError(s, e)
elasticsearch.exceptions.SerializationError: ('{"some-real-json-i-expect ..." } occured while trying to proxy to: materials-es-proxy.takeoffs.io/allproductprices/_search', JSONDecodeError('Unterminated string starting at: line 1 column 7823 (char 7822)'))

It’s worth noting when I run the AWS lambda proxy locally, it works fine, which sort of leads me to believe it’s related to something in AWS, still working through this with them too. The reason I still think it may be related to elasticsearch-py library is because hitting the AWS proxy with postman, curl and even the python requests library, it works fine.

Steps to reproduce: Obviously, it’s not trivial to mock up all the parts for reproducing it. However, we made a temp api key to the proxy that should allow you direct access and direct copy of what I’m doing. Please let me know what you think about this and I’m happy to share it. Here’s a script, you can just fill in the “x-api-key” value once I share it.

from elasticsearch import Elasticsearch

es = Elasticsearch(
    '',
    headers = {
        'x-api-key': '',
    }
)


body = {
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "vendor_id": 103
                    }
                }
            ]
        }
    },
    "aggs": {
		"category.keyword": {
			"terms": {
				"field": "category.keyword",
				"size": 1000,
				"order": {
					"_key": "asc"
				}
			}
		}
	}
}


if __name__ == '__main__':

    with_es = es.search(index='allproductprices', body=body)
    print('RESPONSE', with_es)

[lslee714]

That’s the thing, it doesn’t error and I have a valid dictionary with it and my downstream parsing of it works

        #TODO try and see if we can use the elasticsearch client, currently getting malformed response from aws
        #This is a hack to move forward for now
        res =  requests.post(f"{elastisearch_proxy}/{self.INDEX_NAME}/_search", headers={
            'x-api-key': api_key
        }, json=categories_query)
        raw_result = res.json()
        buckets = raw_result['aggregations']['category.keyword']['buckets']
        ... do the parsing ...

I submitted the logs of it I have to the open ticket on AWS I made earlier to see if they find anything.

[lslee714]

Hey @sethmlarson just FYI we’re still seeing this issue sporadically and AWS support was not able to help.

We moved past the issue by directly using the requests.post method and calling json() on the response, and that works every time. Is the elasticsearch-py library a different deserializer than the requests library? Could be something to look into…

I just did a little digging and it looks like the requests library uses complexjson where as elasticsearch-py uses simplejson