populate event.dataset to allow for dedicated partitions in "Log Rate" ML job
See original GitHub issueThe Log Rate component of the Logs UI sets up an ML job to look for anomalies in log rate counts by event.dataset
. For logs formatted via ecs-logging-java
, that field is not set, so the logs show as “unknown” in the Log Rate UI and are grouped together with all other log sources using ecs-logging-java
, which removes the out-of-the-box ability to see if one source has an unusual amount of logs.
Issue Analytics
- State:
- Created 4 years ago
- Comments:13
Top Results From Across the Web
Creating partitioned tables | BigQuery - Google Cloud
In the Google Cloud console, go to the BigQuery page. Go to BigQuery · In the Explorer pane, expand your project, and then...
Read more >Scaling with Event Hubs - Azure
This article provides information on how to scale Azure Event Hubs by using partitions and throughput units.
Read more >[Logs UI] Include the dataset information in categorization ...
If the job has per-partition categorization enabled, query the categorizer stats document for jobs with categorization status warn (as @ ...
Read more >Serverless Architecture | World of BigData - WordPress.com
In this blog we will talk about how we can implement a batch job using AWS Glue to transform our logs data in...
Read more >Best practices for consuming Amazon Kinesis Data Streams ...
Therefore, the following use cases are challenging for Lambda stream processing: Correlation of events of different shards; Stateful stream ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Having seen a demo of the log categorization, it seems like
${service.name}.log
(for examplemy-application.log
) would be a good default. There should also be a first class citizen configuration for the dataset like this:@felixbarny LGTM.
For doc purposes, the groovy config for this (and the encoder) looks like: