Exception stack trace logs getting split into multiple lines (events) when seen on splunk

What happened: We have been migrating application from EC2 based deployments to kubernetes based deployments. We are seeing the logs get split when we view on splunk. Example a java exception stack trace is multiple events on splunk. This never used to happen previously when we had it hosted on ec2(not containers) What you expected to happen: Entire exception stack trace should be part of single log on splunk How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: This is on EKS with Docker. Environment:

• Kubernetes version (use kubectl version): we are on k8s 1.14
• Ruby version (use ruby --version):
• OS (e.g: cat /etc/os-release): Amazon Linux 2
• Splunk version:
• Others:

[taskar82]

thanks @matthewmodestino. I think we have found our issue now. Also picked up the chat in Slack regarding the timestamp parsing:

time_format was slightly different than SCK though, we were using time_format %Y-%m-%dT%H:%M:%S.%NZ so we switched to keeping time and removing time_format added a filter using the filter_with_time method with essentially

time = Fluent::EventTime.from_time(Time.iso8601(record[‘time’])) record.delete(‘time’)

it reduced CPU utilization quite a bit and improved our throughput

[matthewmodestino]

@rockb1017 have you seen that thread where the user was talking about the differences in performance when doing timestamp parsing?