HEC returns 400 {"text":"Error in handling indexed fields","code":15} to metrics-agg

What happened:

HEC returns 400 Bad Request - {“text”:“Error in handling indexed fields”,“code”:15} to metrics-aggregator

What you expected to happen:

HEC return 200

How to reproduce it (as minimally and precisely as possible):

Deploy Splunk Connect for Kubernetes using Helm chart with mostly defaults (aside from a few changes to customise eg cluster_name) to a Rancher cluster hosted in AWS EC2, enable logging, metrics and objects, observe error in metrics-agg pod.

Anything else we need to know?:

There are many, many log lines being sent with this error.

Here is some output with debug on:

Failed POST to https://xxxxxxxx/services/collector, response: {"text":"Error in handling indexed fields","code":15}
Failed request body: {"host":"","time":"1557389128.8188767","event":"metric","index":"kubernetes_events","source":"source","fields":{"metric_name":"kube.container.memory.request","_value":0.0,"name":"ecr-creds-updater-job","image":"xxxxxxxx","node":"xxxxxxxx","cluster_name":"sandbox","source":"xxxxxxxx"}}

I have tested using CURL to post the failed request body direct to the HEC, modifying the body until it worked. The culprit appears to be the unquoted 0.0 value in “_value”:0.0. When I modify the body to quote this value, the post succeeds and HEC returns 200 OK.

Environment:

• Kubernetes version (use kubectl version): v1.13.5-rancher1-2
• OS (e.g: cat /etc/os-release): RHEL 7.6
• Splunk version: 7.0.1
• Others: Rancher v2.2.2

[TechnicalMercenary]

ok, I just saw this too when going back through documentation etc.

That’s affirmative. I’ll contact our support and see what it’s going to take to upgrade.

Thanks

[chaitanyaphalak]

Hey @TechnicalMercenary its seems you are using eks, we fully support eks. I observed you are using Splunk Version 6.6.3.2, I dont think that version of Splunk has metrics support. If you change the version on this page https://docs.splunk.com/Documentation/Splunk/7.3.1/Metrics/GetStarted to 6.6.3 nothing shows up, suggesting metrics support was added later. Can you confirm this?