Python 3.5.1 - elasticsearch-2.3.0 - RequestError 'failed to parse'

I’m having trouble importing a JSON file into Elasticsearch.

The JSON file is 51 MB, and works when imported into MongoDB

I was following this post https://bitquabit.com/post/having-fun-python-and-elasticsearch-part-1/

Python and Elasticsearch:

Python 3.5.1+ (default, Mar 30 2016, 22:46:26)
[GCC 5.3.1 20160330] on linux
Linux portal-python 4.4.0-16-generic #32-Ubuntu SMP Thu Mar 24 22:38:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Elasticsearch server:

python@portal-python:~$ curl -XGET http://10.0.2.149:9200?pretty
{
  "name" : "Illyana Rasputin",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "5.0.0-alpha1",
    "build_hash" : "7d4ed5b",
    "build_date" : "2016-04-04T10:39:25.841Z",
    "build_snapshot" : false,
    "lucene_version" : "6.0.0"
  },
  "tagline" : "You Know, for Search"
}

Error:

POST /l/data_feed [status:400 request:0.486s]
Traceback (most recent call last):
  File "data_json/python-test.py", line 40, in <module>
    main()
  File "data_json/python-test.py", line 36, in main
    es.index(index='kl', doc_type=ES_TYPE, body=data)
  File "/home/python/virtualenv/elasticsearch/lib/python3.5/site-packages/elasticsearch-2.3.0-py3.5.egg/elasticsearch/client/utils.py", line 69, in _wrapped
  File "/home/python/virtualenv/elasticsearch/lib/python3.5/site-packages/elasticsearch-2.3.0-py3.5.egg/elasticsearch/client/__init__.py", line 279, in index
  File "/home/python/virtualenv/elasticsearch/lib/python3.5/site-packages/elasticsearch-2.3.0-py3.5.egg/elasticsearch/transport.py", line 329, in perform_request
  File "/home/python/virtualenv/elasticsearch/lib/python3.5/site-packages/elasticsearch-2.3.0-py3.5.egg/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
  File "/home/python/virtualenv/elasticsearch/lib/python3.5/site-packages/elasticsearch-2.3.0-py3.5.egg/elasticsearch/connection/base.py", line 108, in _raise_error
elasticsearch.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'failed to parse')

Python:

#!/usr/bin/env python
# -*- coding: utf8 -*-

from datetime import datetime
import elasticsearch
import json
import os

es_server = 'http://10.0.2.149:9200'
es = elasticsearch.Elasticsearch([es_server])

script_path = os.path.realpath(__file__)
working_path = os.path.split(script_path)[0]


def main():
    ES_TYPE = 'data_feed'
    ES_INDEX = 'l'

    for path, subdirs, files in os.walk(working_path):

        for file in files:
            path_file = os.path.join(path, file)

            if os.path.isfile(path_file):

                if os.path.splitext(path_file)[1] == '.json':
                    # ES_INDEX = os.path.splitext(file)[0]

                    with open(path_file, 'r') as f:
                        data = json.load(f)

                    print('Importing:', file, '(', int(os.stat(path_file).st_size / 1024 / 1024), 'MB', ')')
                    es.index(index='l', doc_type=ES_TYPE, body=data)


if __name__ == "__main__":
    main()

[honzakral]

Thanks! The issue is that documents need to be json objects where you are trying to use a list.

If you want to index all documents from that list you need another for loop to iterate over them or use something like the bulk helper to index multiple documents at the same time.

http://elasticsearch-py.readthedocs.org/en/master/helpers.html

[honzakral]

Definitely use the bulk api via the helper.