Elasticsearch could fail to ingest output from esedb/srum

Description of problem:

When uploading data to elasticsearch/timesketch via psort.py, elastic could fail to ingest most of esedb/srum event ( and also windows:registry:winlogon event). This case append when the first srum event send to elastic have an application en user_identifier that were not resolved against the SruDbIdMapTable. In this case elasticsearch map internally the user_identifier and application field as long integer and failed to ingest data where these field are text.

Command line and arguments:

Source data:

could not provide

Plaso version:

For example 20200717

Operating system Plaso is running on:

Ubuntu

Installation method:

• installed from [GiFT PPA][https://launchpad.net/~gift] stable track

This issue could be resolved by two different way (I have tested both) :

• forcing the mapping of these to field when the index is created { "properties": { "application": { "type": "text", "fields": {"keyword": {"type": "keyword"}}, }, "user_identifier": { "type": "text", "fields": {"keyword": {"type": "keyword"}}, }, } }
• just cast these value to string when they are created : ( replace line 333 of https://github.com/log2timeline/plaso/blob/master/plaso/parsers/esedb_plugins/srum.py : record_value = identifier_mappings.get(record_value, record_value) -> record_value = str(identifier_mappings.get(record_value, record_value))

Imo the second option is the best but I let you see which one you prefer

[joachimmetz]

@william-billaud thx, I’ll have closer look when time permits. @kiddinn who indicated going to work on a new TS and ES output module https://github.com/log2timeline/plaso/issues/3097

[kiddinn]

thank you for this, I’ll take a closer look at this shortly, since I will be re-designing the ES and TS outputs