Kubernetes logs in the "default" namespace have the namespace value "main"

I’ve pinpointed where the issue is although I’m wary of making changes myself as I don’t know why the line was introduced in the first place. Skip to “Anything else we need to know?” for the root cause.

What happened: Kubernetes logs in the “default” namespace are being reported as being from the namespace “main” in Splunk.

What you expected to happen: Kubernetes logs in the “default” namespace should be reported as being from the “default” namespace in Splunk.

How to reproduce it (as minimally and precisely as possible):

1. Set up SCK with a pod in the “default” namespace
2. Look at the “namespace” field for the logs for that pod on Splunk

Anything else we need to know?: There was a commit to support index routing [1]. This added a feature where logs could be forwarded to an index with the same name as the log’s namespace. The default namespace for logs is “default”. The default index for Splunk is “main”. There was logic added to make sure logs from “default” namespace go into the “main” index [2]. However, in a separate file [3], the namespace field is also converted from “default” to “main”.

def set_namespace(value): if value == \"default\" then \"main\"else value end;

This seems very wrong as it breaks log correlation.

You can contact me over Slack as I am a Splunk employee.

[1] https://github.com/splunk/splunk-connect-for-kubernetes/commit/96df23c52c4af454d3671b92682f8dd8f8a03acc [2] https://github.com/splunk/splunk-connect-for-kubernetes/commit/96df23c52c4af454d3671b92682f8dd8f8a03acc#diff-dcb43995d0de88c51cf7a3d031a436dc [3] https://github.com/splunk/splunk-connect-for-kubernetes/blob/06e877daa21cff4420ededd069e20774e5b584d8/manifests/splunk-kubernetes-logging/configMap.yaml#L182

Environment:

• Kubernetes version (use kubectl version): 1.13.4
• OS (e.g: cat /etc/os-release): MacOS High Sierra
• Splunk version: 7.3.0

[hexecute]

“if you dont want the data from “default” namespace to route to “main” index” This isn’t about routing but data integrity. The logic for routing shouldn’t be modifying the logs data to achieve the desired behavior.

Your suggestion is a good workaround (indexRouting = false and indexRoutingDefaultIndex = default), and we’ll use that in our setup for now. I might create a PR in my spare time later.

[chaitanyaphalak]

This issue is addressed in https://github.com/splunk/splunk-connect-for-kubernetes/pull/215