Easy way to bulk index a JSON Document?

Hi, is there an easy way to bulk index a normal JSON document rather than formatting data to look like

[
    // action description
    { index:  { _index: 'myindex', _type: 'mytype', _id: 1 } },
     // the document to index
    { title: 'foo' },
    // action description
    { update: { _index: 'myindex', _type: 'mytype', _id: 2 } },
    // the document to update
    { doc: { title: 'foo' } },
    ...
]

as is shown in the elasticsearch-jsAPI Docs?

So basically I’m trying to omit the action description object, so my body would look like

[ 
   { data_source_1 },
   { data_source_2 },
   ...
]

I don’t mind the _id being auto generated (unless that has some negative consequences that I’m not aware of). I thought the bulk method also takes in the index and type in the parameters so it seems redundant to specify those for each document if you’re just indexing one type in one index. I guess you do still need the action verb, but it would be nice if you could specify that in the parameters as well.

I feel like a lot of JSON data out there doesn’t have the action description object, and so would have to be added after, which seems like a painful task. What are people doing when they have to index lots of documents at once?

[samechikson]

Hi @adilld , es-json-load loads a normal JSON array directly into Elasticsearch. It does not generate a file which you can then use with the bulk API, which is what you seem to want to do.

In terms of usage, if you install it globally from npm (npm install es-json-load -g), then you should be able to use it in the same way as a normal binary. Something like this

es-json-load --data --file=/absolute/path/to/file --index=<index-name> --type=<type-name>

Make sense?

[samechikson]

@adilld Sounds like you’re trying pipe data into ES, so consider using Logstash which is a dedicated tool written by Elastic for this. You can use CSV files directly.

If you are to use es-json-load, transforming a CSV file into a JSON file should work. Just make sure that the JSON file that you use is a JSON Array of objects. This would look something like [{},{},{},...]