Elasticsearch output plugin - Errors on _bulk api post

Environment

• How did you install GRR? [e.g. from release DEB, HEAD DEB, PIP package, source] Dockerfile - https://hub.docker.com/layers/grr/grrdocker/grr/v3.4.6.0/images/sha256-30f404087124417eb96420ad157c5bac403748c55eafabf1e0abfbf7c20bb0c9?context=explore

• What GRR version are you running?: [e.g. 3.1.2.3] 3.4.6.0

• What operating system does the GRR server run on? [e.g. Ubuntu 18.04] Ubuntu 18.04. This is running in aks (Azure kubernetes service) currently using image-name: “AKSUbuntu-1804gen2containerd-2022.06.08”

• What operating system does the affected GRR client run on, if applicable? [e.g. Windows 10] Windows server 2022 (Not affected)

• Elastic cluster version (Likely of relevance here) Im using a single node elasticsearch cluster deployed as image (container) )version 7.17.0 deployed with the elastic helmchart elastic/elasticsearch.

Describe the issue The outputplugin elasticsearch throws errors on usage. In this case im creating a flow to request the netstat listenting ports/processes on a windows based client. API request for the flow looks like the following (Where “grr-server-host” is a dummy replacement for a valid url).

CSRFTOKEN=`curl https://grr-server-host -o /dev/null -s -c - | grep csrftoken  | cut -f 7`; \
	curl -X POST -H "Content-Type: application/json" -H "X-CSRFToken: $CSRFTOKEN" \
	https://grr-server-host/api/v2/clients/C.7e53019687aed8b7/flows -d @- << EOF
{
  "flow": {
    "args": {
      "@type": "type.googleapis.com/grr.NetstatArgs",
      "listeningOnly": true
    },
    "name": "Netstat",
    "runnerArgs": {
      "outputPlugins": [
        {
          "pluginName": "ElasticsearchOutputPlugin",
          "pluginArgs": "CgduZXRzdGF0CgNncnI="
        }
      ]
    }
  }
}
EOF

Without the outputplugin it all works fine.

Its when i add the outputplugin i start getting issues in the grr-worker. This is shown in the grr-worker.log. This is visible both within the pod/container running grr_worker and the admin_ui when viewing a specific flow under the navbar item “managed launched flows” -> “results” -> “1 errors” (clicking this opens a modal view of an errors list with the errors below).

!OBS See Debugging process blow.

• “Error while processing 23 replies: 406 Client Error: Not Acceptable for url: https://grr-server-host:9200/_bulk” Upon inspection this appears to be due to a default of a "content type" : "url encode" this is resolved by adding contenttype headers for application/json.
• “Error while processing 23 replies: 400 Client Error: Bad Request for url: https://grr-server-host:9200/_bulk” Upon inspection this is due to a missing new line at the end of the json request.

Error logs

From grr-worker.log

(grr-server) root@grr-6c6c959d75-vs9w9:/usr/share/grr-server# grr_worker --verbosity 1 --alsologtostderr
I0712 14:20:16.563979 139940397143872 config_lib.py:870] Loading configuration from /usr/share/grr-server/install_data/etc/grr-server.yaml
I0712 14:20:16.601257 139940397143872 config_lib.py:404] Applying filter resource for install_data/etc@grr-response-core.
I0712 14:20:16.602211 139940397143872 config_lib.py:870] Loading configuration from /usr/share/grr-server/install_data/etc/server.local.yaml
I0712 14:20:16.623253 139940397143872 config_lib.py:576] Configuration writeback is set to /usr/share/grr-server/install_data/etc/server.local.yaml
I0712 14:20:16.624943 139940397143872 server_logging.py:205] Initializing Logging subsystem.
I0712 14:20:16.625142 139940397143872 server_logging.py:158] Will use logging engines ['stderr', 'file']
I0712 14:20:16.625748 139940397143872 config_lib.py:404] Applying filter resource for grr_response_core@grr-response-core.
I0712 14:20:16.626402 139940397143872 server_logging.py:187] Writing log file to /usr/share/grr-server/lib/python3.6/site-packages/grr_response_core/var/log//grr-worker.log
ERROR:2022-07-12 12:55:05,629 386 MainProcess 140000134067968 flow_processing_pool-252777379 flow_base:920] Plugin ElasticsearchOutputPlugin <<ElasticsearchOutputPluginArgs("message ElasticsearchOutputPluginArgs {\n tags : [\n   'netstat'\n   'grr'\n  ]\n}")>> failed to process 23 replies.
Traceback (most recent call last):
  File "/usr/share/grr-server/lib/python3.6/site-packages/grr_response_server/flow_base.py", line 900, in _ProcessRepliesWithFlowOutputPlugins
    [r.AsLegacyGrrMessage() for r in replies])
  File "/usr/share/grr-server/lib/python3.6/site-packages/grr_response_server/output_plugins/elasticsearch_plugin.py", line 89, in ProcessResponses
    self._SendEvents(events)
  File "/usr/share/grr-server/lib/python3.6/site-packages/grr_response_server/output_plugins/elasticsearch_plugin.py", line 156, in _SendEvents
    response.raise_for_status()
  File "/usr/share/grr-server/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://grr-server-host:9200/_bulk

Additional context Debugging Process In my case while testing and developing a fix for this i was using a small test.jsonl file generated with real data extracted by adding an exception to the plugin to dump the expected body of the request. I added the line raise Exception("CheckJsonFormat: ---%s",data) at lin 150 in the plugin. https://github.com/google/grr/blob/master/grr/server/grr_response_server/output_plugins/elasticsearch_plugin.py#L150

Here is my output file:

{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 445},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 5985},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 47001},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "192.168.1.5","port": 139},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 5985},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 445},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 47001},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49666},"state": "LISTEN","pid": 420,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49666},"state": "LISTEN","pid": 420,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49665},"state": "LISTEN","pid": 568,"processName": "wininit.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49665},"state": "LISTEN","pid": 568,"processName": "wininit.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49669},"state": "LISTEN","pid": 640,"processName": "services.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49669},"state": "LISTEN","pid": 640,"processName": "services.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49664},"state": "LISTEN","pid": 648,"processName": "lsass.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49664},"state": "LISTEN","pid": 648,"processName": "lsass.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 3389},"state": "LISTEN","pid": 708,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 3389},"state": "LISTEN","pid": 708,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 135},"state": "LISTEN","pid": 836,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 135},"state": "LISTEN","pid": 836,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49667},"state": "LISTEN","pid": 1016,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49667},"state": "LISTEN","pid": 1016,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49668},"state": "LISTEN","pid": 1848,"processName": "spoolsv.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49668},"state": "LISTEN","pid": 1848,"processName": "spoolsv.exe"},"tags": ["netstat","grr"]}

Then using this file with curl to attempt the connections myself…

1. port-forward the elasticsearch server via kubectl
2. replicate the request in the form used by the elasticsearch plugin
3. get more human friendly errors.

The first curl request looks as follows, where test.jsonl is the json extracted from the request as mentioned above.

curl -H "Authorization: Basic <auth-token-here>" "https://localhost:9200/_bulk" -k -XPOST --data-binary "@test.jsonl"

This results in the error {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406} So to fix this we add content type to the request and try again.

curl -H "Authorization: Basic <auth-token-here>" "https://localhost:9200/_bulk" -k -XPOST --data-binary "@test.jsonl" -H "Content-Type: application/json"

This results in the error {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"},"status":400}.

To fix this we add a \n at the end of the json body.

[max-vogler]

Thanks for reporting and debugging this issue! Please send a pull request with your fix.

[micrictor]

Original dev of the Elasticsearch output plugin here.

I looked back at docs for old versions, I don’t think there’s a regression risk as all versions that still have live docs include that header value.

I submitted my changes via PR, so that seems like a decent route. If you want, I can incorporate your changes and add the requisite tests to ensure that my (broken )😃 code would have failed.