AWS Lambda Log forwarder: Metadata should be unique per CloudWatch log item

Describe what happened:

awslogs_handler() (and maybe other functions there, I didn’t check) fetches several logs from one CloudWatch event in this code

    with gzip.GzipFile(
        fileobj=BytesIO(base64.b64decode(event["awslogs"]["data"]))
    ) as decompress_stream:
        data = b"".join(BufferedReader(decompress_stream))
    logs = json.loads(data)

After this it fills metadata with values using rules based on source detection:

metadata[DD_SOURCE] = parse_event_source(event, source)
# ...
if metadata[DD_SOURCE] == "cloudwatch":
    metadata[DD_HOST] = aws_attributes["aws"]["awslogs"]["logGroup"]
# ...
if metadata[DD_SOURCE] == "rds":
# ...
# etc.

The problem is that in some cases log messages contain information about the real source host, “service” and also log severity.

The severity detection logic (metatada['status']) is not presented in the forwarder currently, but it remains beyond the goal of this issue.

With current implementation, all logs from one bulk will get the same metadata which will be merged into the final object in parse().

def parse(event, context):
    # ...
        elif event_type == "awslogs":
            events = awslogs_handler(event, context, metadata)
    # ...
    return normalize_events(events, metadata)

And technically, the normalize_events(events, metadata) adds a copy of the metadata into each individual “event” (read “log item”).

Describe what you expected:

Each “event” inside events should have its own unique copy of metadata and all tags (keys) of the metadata, including DD_SOURCE, DD_HOST, DD_SERVICE (and DD_STATUS in future implementations) and all metadata tags should be potentially evaluated per log line, not per events.

More practically, the metadata should be changeable within the loop:

https://github.com/DataDog/datadog-serverless-functions/blob/f7568f43efa343b9fe3c2c5da5791767d8c20d5d/aws/logs_monitoring/lambda_function.py#L826

Current version:

    # Create and send structured logs to Datadog
    for log in logs["logEvents"]:
        yield merge_dicts(log, aws_attributes)

Potential to change in the future:

    # Create and send structured logs to Datadog
    for log in logs["logEvents"]:
        if metadata[DD_SOURCE] == "kubernetes":
            # the metadata must be unique, right now this does not work
            set_k8s_metadata(log, metadata)
        yield merge_dicts(log, aws_attributes)

Steps to reproduce the issue:

Try to set metadata['status'] (severity) based on a specific log line. All logs in the batch will get the status of the latest log line in the batch.

[rnegron]

I think this is probably the issue I’m having or at least related to it… Let me know if I’m mistaken and I’ll go ahead and open a new issue so as not to derail this one.

So, I’m trying to use the Lambda Log Forwarder with a few Cloudwatch log groups as sources, and those log group contain the logs of a web application and some workers, all running on ECS Fargate. The thing is, in DataDog, the service gets reported as being fargate in all cases, probably due to the following line: https://github.com/DataDog/datadog-serverless-functions/blob/master/aws/logs_monitoring/lambda_function.py#L913

It seems to find that the log group has /fargate in its name and reports it as the source. But, I am using log injection as specified in the DataDog logs so that the actual log object is a JSON that contains dd.service, dd.env and so on. The dd.service vaue is being ignored here as I understand it.

[KIVagant]

For normal use case – consuming logs from CW log groups, we do expect events in the same batch to share the same metadata.

With all respect, I would really vote against this approach. In our case, for instance, we send logs from hundreds of applications to a log group. I believe any random cloudwatch log group may contain logs from different hosts, apps etc. For example, even if we simplify the use case, let’s imagine we have a log group that contains logs from 10 different hosts. And log lines have the host ID inside. To configure metadata[DD_HOST] properly, we need to parse each log line and extract the information from it (actually there can be many other important fields inside). With current implementation this is not possible. The DD_HOST will be set to logs["logGroup"] which does not make big sense.

And there are many other use cases when metadata should be unique. For instance, an extended logs filtration based on the grabbed metadata information. Currently there are only EXCLUDE_AT_MATCH and INCLUDE_AT_MATCH available for this purpose. But it’s really, really not enough for big production. Taking into account that DataDog charges for all incoming logs even if the main index has a lot of exclusion filters, we need to move the exclusion logic to the Lambda function.

You could face the same issue if you tried to parse logs collected from a Kubernetes cluster, streamed with, let’s say, Fluentd-to-CloudWatch.