further .text and keyword discussion

for security use cases, large error messages, etc… .text is an important aspect in searching.

pro(s):

• faster searches
• severely reduces impact on the backend (and ultimately customer/user experience)

con(s):

• increases storage

temporary proposal to meet in the middle (for now) instead of adding .text into the dynamic template for all strings fields VS having no .text fields at all ECS and thus corresponding beats index templates have .text field for high value/impacting fields. List of fields off the top of my head would be:

• process.executable.text
• process.args.text
• url.original
• user.email
• user_agent.original
• error.stack_trace
• file.path
• host.name
• any .user.name fields
• any .domain fields
• any .as.organization.name fields
• http.request.body.content
• http.response.body.content
• os.name.full

existing ECS reference: https://github.com/elastic/ecs/issues/340 https://github.com/elastic/ecs/issues/104

[randomuserid]

So I reviewed all of our current rule sets and there are a few more ECS fields that are used by hunting searches. This list is not ranked but the process fields are clearly the most heavily used and process.args / process.command_line may be the most frequently used. These are the fields not listed above:

process.command_line process.name process.parent.args process.parent.executable process.pe.original_file_name process.pe.signer_name process.working_directory

There are also 13 non-ECS fields that are used by hunting searches or by ML job datafeed queries in addition to 12 ECS fields for a total of 25. The complete breakdown is in here: https://docs.google.com/spreadsheets/d/1xOicLbct4Vk10hj4_Xaa1ViYjU9W_unFZ_Rv_mTe-ks/edit#gid=0

[webmat]

I would add os.name.full to the list. See why in https://github.com/elastic/ecs/issues/576#issuecomment-540748059