question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

populate event.dataset to allow for dedicated partitions in "Log Rate" ML job

See original GitHub issue

The Log Rate component of the Logs UI sets up an ML job to look for anomalies in log rate counts by event.dataset. For logs formatted via ecs-logging-java, that field is not set, so the logs show as “unknown” in the Log Rate UI and are grouped together with all other log sources using ecs-logging-java, which removes the out-of-the-box ability to see if one source has an unusual amount of logs.

image

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:13

github_iconTop GitHub Comments

1reaction
felixbarnycommented, Feb 12, 2020

Having seen a demo of the log categorization, it seems like ${service.name}.log (for example my-application.log) would be a good default. There should also be a first class citizen configuration for the dataset like this:

    encoder(EcsEncoder) {
        serviceName = "my-service"
        eventDataset = "my-service.logins"
    }
0reactions
mbarrettacommented, Feb 10, 2020

@felixbarny LGTM.

For doc purposes, the groovy config for this (and the encoder) looks like:

    encoder(EcsEncoder) {
        serviceName = "my-service"
        additionalField(EcsEncoder.Pair) {
            key="event.dataset"
            value="my-service"
        }
    }
Read more comments on GitHub >

github_iconTop Results From Across the Web

Creating partitioned tables | BigQuery - Google Cloud
In the Google Cloud console, go to the BigQuery page. Go to BigQuery · In the Explorer pane, expand your project, and then...
Read more >
Scaling with Event Hubs - Azure
This article provides information on how to scale Azure Event Hubs by using partitions and throughput units.
Read more >
[Logs UI] Include the dataset information in categorization ...
If the job has per-partition categorization enabled, query the categorizer stats document for jobs with categorization status warn (as @ ...
Read more >
Serverless Architecture | World of BigData - WordPress.com
In this blog we will talk about how we can implement a batch job using AWS Glue to transform our logs data in...
Read more >
Best practices for consuming Amazon Kinesis Data Streams ...
Therefore, the following use cases are challenging for Lambda stream processing: Correlation of events of different shards; Stateful stream ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found