"400 Bad Request Event field cannot be blank" when empty line is logged

I am running latest version of logging module of splunk connect for k8s (1.0.1).

Recently I have stumbled on a strange issue, whole batches of logs (fluentd tries to send logs in batches) were discarded by HEC with 400 response and message Bad Request Event field cannot be blank. Looks like all the processing done with jq transforms new line only log messages (\n) to empty events. HEC drops the whole batch even if the only one message it empty. In result, we were missing huge chunks of the logs. I know it is partially due to bad logging on our side. However, I think fluentd/Splunk should do a better job to protect the operator against such issues. As a workaround I have added:

      # ensure we do not have empty line logs, they cannot be ingested by Splunk and result in 400 response from
      # the Splunk HEC
      <filter tail.containers.**>
        @type jq_transformer
        jq 'if .record.log == "\n" then .record.log = "E" else .record.log = .record.log end | .record'
      </filter>

right before <filter tail.containers.**>. It does the job, we do not lose logs anymore, developers see they application is pushing garbage (empty logs are shown as E in Splunk).

Is there a better way of solving this?

[matthewmodestino]

@rockb1017 @chaitanyaphalak can we revisit this? Ran into it again today. Should have a safeguard in the filters that protects against chunk loss due to empty events.

For now I’m going to add this at the end. Thanks @h0ppyf33t

 # filter to  remove empty lines
      <filter tail.containers.**>
        @type grep
        <exclude>
         key log
         pattern ^$
        </exclude>
      </filter>

And try to ensure a concat filter is added to any offenders

Thanks!

Matt

[dancb10]

@szymonpk following your advice I’ve forked the helm and added the following in the configMap.yaml, in the output.conf section:

      {{- if and .Values.customFilters }}
      {{- range $name, $filterDef := .Values.customFilters }}
      <filter {{ $filterDef.tag }}>
        @type {{ $filterDef.type }}
        {{ $filterDef.body }}
      </filter>
      {{- end }}
      {{- end }}

The section is placed just above the section ‘# extract index fields and sourcetype for container logs <filter tail.containers.**>’ In the values.yaml file I’ve defined a default filter as follows:

customFilters:
  EmptyEventFilter:
    tag: tail.containers.**
    type: jq_transformer
    body: >
        jq 'if .record.log == "\n" then .record.log = "EmptyEvent" else .record.log = .record.log end | .record'

So now all events with the empty field can be seen under the “EmptyEvent” section