splunk-kubernetes-logging - allow pod and container exclusion

Hi everybody,

first off: Great work. This is really easy to use!

What would you like to be added:

I’d like to be able to limit the collected pod logs, in order to limit what is send to Splunk. Currently I am doing the following:

global:
  fluentd:
    # path of logfiles, default /var/log/containers/*.log
    path: /var/log/containers/*.log
    # paths of logfiles to exclude. object type is array as per fluentd specification:
    # https://docs.fluentd.org/input/tail#exclude_path
    exclude_path:
     - /var/log/containers/prometheus*.log
     - /var/log/containers/istio*.log
     - /var/log/containers/kube*.log

This works if I want to exclude pods (even though it is cumbersome), but this won’t affect side car containers. So for Istio for example, I might only be interested in the logs of my apps, but not of the istio-proxy sidecar.

Why is this needed:

Limit the input into Splunk and increase visibility

[chaitanyaphalak]

Hey @stiller-leser, thanks for the kudos. I think that you might be able to add an exclude path for the pods you dont want. For example, if your app’s pod name is “my-awesome-pod-xxxxxxx” I think you would be able to ignore logs for that using “/var/log/containers/my-awesome-pod*.log”. I agree that this approach is cumbersome. But I think if you really want it, you could do a pull request for us in the following way: Approach A)

• In values.yaml file take a list of pod names you want to ignore
• Then add a function to https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/helm-chart/splunk-kubernetes-logging/templates/_helpers.tpl file to check for pod_name and check if its in the ignore list, if yes drop the record.

Approach B)

• use a custom filter https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/helm-chart/splunk-kubernetes-logging/values.yaml#L355 to drop the records using a similar logic/using another plugin.

Please try out some other ideas too, if you want and let me know if you need help. I would be happy to review a PR for the implementation.

Thanks, Happy Friday.

[PrabuddhaRaj]

HI @chaitanyaphalak : i am using exclude_path to remove logs of the sidecar pod running in my kubernetes cluster. But it doesnt work. i tried annotating the pod with splunk.com/exclude=true but that also doesnt work as i am facing issue in restarting the pod and secondly i am suing kubectl to annotate. it should be done via script such values.yaml file. I only want logs of my application pod any help is appreciated