question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Bug] Not every cloudwatch message is passed

See original GitHub issue

Describe what happened: We have noticed that for very very long messages that are stored in Cloudwatch, log forwarder is not passing all of them.

I have done some tests and found that Cloudwatch is automatically splitting longer lines to the size of 262119 characters. the maximum that logs forwarder is accepting is 256000 which causes some of logs to be dropped.

The difference is not big and actually changing the DatadogBatcher max_log_size_bytes parameter to 265000 did the trick. And all logs from Cloudwatch was passed further.

I’m not sure if there are any other consequences of this change.

Describe what you expected:

All entries stored in Cloudwatch should be passed to Datadog.

Steps to reproduce the issue:

Create a log entry in Cloudwatch that is between 256000 and 262119 characters.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
tianchucommented, Aug 10, 2021

I can reproduce it and apparently CloudWatch logs’ documented event size limit is 256K, but in reality it’s 262,144 (can be found when attempting to create a big event in the CW logs console). After discussion with our logs intake team, it is safe to bump the event size limit. Considering the metadata and other overhead, I will bump the limit to 512K (the backend limit is 1M).

1reaction
tianchucommented, Aug 9, 2021

@radekl Thanks for letting us know, I’m going to do some experiment too. Really appreciate your feedback!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting and known issues - Amazon CloudWatch
The first step to troubleshooting any issues is to enable debug logging on the Lambda Insights extension. To do this, set the following...
Read more >
CloudWatch Insights: get logs of errored lambdas
If someone comes here looking for a solution, here's what I use: filter @message like /(?i)(Exception|error|fail)/| fields @timestamp, ...
Read more >
Filter and pattern syntax - Amazon CloudWatch Logs
The following code snippet shows an example of a filter pattern that returns all log events where messages contain the word ERROR or...
Read more >
The Guide To CloudWatch Insights With Instructions & Examples
Understand how to use CloudWatch Insights to ingest, index, and query data from system logs and learn its limitations and best practices.
Read more >
Saving on AWS Lambda Amazon CloudWatch Logs costs
The service is not all sunshine and rainbows – things tend to fail. ... releases all log.debug messages, along with the log.error message....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found