AWS Lambda Log forwarder: Metadata should be unique per CloudWatch log item
See original GitHub issueDescribe what happened:
awslogs_handler()
(and maybe other functions there, I didn’t check) fetches several logs from one CloudWatch event in this code
with gzip.GzipFile(
fileobj=BytesIO(base64.b64decode(event["awslogs"]["data"]))
) as decompress_stream:
data = b"".join(BufferedReader(decompress_stream))
logs = json.loads(data)
After this it fills metadata
with values using rules based on source detection:
metadata[DD_SOURCE] = parse_event_source(event, source)
# ...
if metadata[DD_SOURCE] == "cloudwatch":
metadata[DD_HOST] = aws_attributes["aws"]["awslogs"]["logGroup"]
# ...
if metadata[DD_SOURCE] == "rds":
# ...
# etc.
The problem is that in some cases log messages contain information about the real source host, “service” and also log severity.
The severity detection logic (
metatada['status']
) is not presented in the forwarder currently, but it remains beyond the goal of this issue.
With current implementation, all logs from one bulk will get the same metadata
which will be merged into the final object in parse()
.
def parse(event, context):
# ...
elif event_type == "awslogs":
events = awslogs_handler(event, context, metadata)
# ...
return normalize_events(events, metadata)
And technically, the normalize_events(events, metadata)
adds a copy of the metadata
into each individual “event” (read “log item”).
Describe what you expected:
Each “event” inside events
should have its own unique copy of metadata
and all tags (keys) of the metadata
, including DD_SOURCE
, DD_HOST
, DD_SERVICE
(and DD_STATUS
in future implementations) and all metadata tags should be potentially evaluated per log line, not per events.
More practically, the metadata
should be changeable within the loop:
Current version:
# Create and send structured logs to Datadog
for log in logs["logEvents"]:
yield merge_dicts(log, aws_attributes)
Potential to change in the future:
# Create and send structured logs to Datadog
for log in logs["logEvents"]:
if metadata[DD_SOURCE] == "kubernetes":
# the metadata must be unique, right now this does not work
set_k8s_metadata(log, metadata)
yield merge_dicts(log, aws_attributes)
Steps to reproduce the issue:
Try to set metadata['status']
(severity) based on a specific log line. All logs in the batch will get the status of the latest log line in the batch.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:2
- Comments:7 (6 by maintainers)
I think this is probably the issue I’m having or at least related to it… Let me know if I’m mistaken and I’ll go ahead and open a new issue so as not to derail this one.
So, I’m trying to use the Lambda Log Forwarder with a few Cloudwatch log groups as sources, and those log group contain the logs of a web application and some workers, all running on ECS Fargate. The thing is, in DataDog, the
service
gets reported as beingfargate
in all cases, probably due to the following line: https://github.com/DataDog/datadog-serverless-functions/blob/master/aws/logs_monitoring/lambda_function.py#L913It seems to find that the log group has
/fargate
in its name and reports it as the source. But, I am using log injection as specified in the DataDog logs so that the actual log object is a JSON that containsdd.service
,dd.env
and so on. Thedd.service
vaue is being ignored here as I understand it.With all respect, I would really vote against this approach. In our case, for instance, we send logs from hundreds of applications to a log group. I believe any random cloudwatch log group may contain logs from different hosts, apps etc. For example, even if we simplify the use case, let’s imagine we have a log group that contains logs from 10 different hosts. And log lines have the host ID inside. To configure
metadata[DD_HOST]
properly, we need to parse each log line and extract the information from it (actually there can be many other important fields inside). With current implementation this is not possible. The DD_HOST will be set tologs["logGroup"]
which does not make big sense.And there are many other use cases when metadata should be unique. For instance, an extended logs filtration based on the grabbed metadata information. Currently there are only
EXCLUDE_AT_MATCH
andINCLUDE_AT_MATCH
available for this purpose. But it’s really, really not enough for big production. Taking into account that DataDog charges for all incoming logs even if the main index has a lot of exclusion filters, we need to move the exclusion logic to the Lambda function.You could face the same issue if you tried to parse logs collected from a Kubernetes cluster, streamed with, let’s say, Fluentd-to-CloudWatch.