Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
KafkaClientMetrics not working as expected (topic names missing etc.)
See original GitHub issueHi all, we are currently upgrading from Spring Boot 2.2.7 to Spring Boot 2.3.1 and therewith we also want to start using the new KafkaClientMetrics using the FactoryListener from Spring Kafka.
However, we currently see that the metrics do not contain all expected tags (like topic names etc.) and that not all topic-related metrics are provided for all topics.
In the example below you can see that we have only 3 metrics for the average lag (with topic names), the metrics for max lag do not contain topic names at all etc.
Any idea why this is the case?
I would have expected to have all topic-related metrics with a topic name as a tag. Another service that we have upgraded to use the new KafkaClientMetrics only shows the max lag metrics (without topic names) and no other lag-related metrics at all-
# HELP kafka_consumer_fetch_manager_records_lag_avg The average lag of the partition
# TYPE kafka_consumer_fetch_manager_records_lag_avg gauge
kafka_consumer_fetch_manager_records_lag_avg{client_id="csm-cloud-project-craft-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-0",topic="csm_local_referencedata_craft",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="csm-cloud-project-1",kafka_version="5.4.2-ccs",partition="1",spring_id="kafkaConsumerFactory.csm-cloud-project-1",topic="csm_local_projectmanagement_project_delete",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="csm-cloud-project-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-0",topic="csm_local_projectmanagement_project_delete",} NaN
# HELP kafka_consumer_fetch_manager_records_lag_max The maximum lag in terms of number of records for any partition in this window
# TYPE kafka_consumer_fetch_manager_records_lag_max gauge
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-0",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-0",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-user-1",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-user-1",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-user-2",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-user-2",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-2",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-2",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-1",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-1",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-craft-0",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-0",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-craft-1",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-1",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-craft-2",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-2",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-user-0",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-user-0",} NaN
# HELP kafka_consumer_fetch_manager_records_lag The latest lag of the partition
# TYPE kafka_consumer_fetch_manager_records_lag gauge
kafka_consumer_fetch_manager_records_lag{client_id="csm-cloud-project-1",kafka_version="5.4.2-ccs",partition="1",spring_id="kafkaConsumerFactory.csm-cloud-project-1",topic="csm_local_projectmanagement_project_delete",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="csm-cloud-project-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-0",topic="csm_local_projectmanagement_project_delete",} 0.0
UPDATE: I updated again to Micrometer 1.5.3 and now the lag metrics are complete. However, the max lag metrics still do not contain the topic name just the regular lag metrics do.
# HELP kafka_consumer_fetch_manager_records_lag The latest lag of the partition
# TYPE kafka_consumer_fetch_manager_records_lag gauge
kafka_consumer_fetch_manager_records_lag{client_id="csm-cloud-project-craft-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-0",topic="csm_local_referencedata_craft",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="csm-cloud-project-user-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-user-0",topic="csm_local_usermanagement_user",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="csm-cloud-project-1",kafka_version="5.4.2-ccs",partition="1",spring_id="kafkaConsumerFactory.csm-cloud-project-1",topic="csm_local_projectmanagement_project_delete",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="csm-cloud-project-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-0",topic="csm_local_projectmanagement_project_delete",} 0.0
# HELP kafka_consumer_fetch_manager_records_lag_max The maximum lag in terms of number of records for any partition in this window
# TYPE kafka_consumer_fetch_manager_records_lag_max gauge
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-0",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-0",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-user-1",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-user-1",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-user-2",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-user-2",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-2",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-2",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-1",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-1",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-craft-0",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-0",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-craft-1",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-1",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-craft-2",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-2",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="csm-cloud-project-user-0",kafka_version="5.4.2-ccs",spring_id="kafkaConsumerFactory.csm-cloud-project-user-0",} NaN
# HELP kafka_consumer_fetch_manager_records_lag_avg The average lag of the partition
# TYPE kafka_consumer_fetch_manager_records_lag_avg gauge
kafka_consumer_fetch_manager_records_lag_avg{client_id="csm-cloud-project-craft-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-craft-0",topic="csm_local_referencedata_craft",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="csm-cloud-project-user-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-user-0",topic="csm_local_usermanagement_user",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="csm-cloud-project-1",kafka_version="5.4.2-ccs",partition="1",spring_id="kafkaConsumerFactory.csm-cloud-project-1",topic="csm_local_projectmanagement_project_delete",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="csm-cloud-project-0",kafka_version="5.4.2-ccs",partition="0",spring_id="kafkaConsumerFactory.csm-cloud-project-0",topic="csm_local_projectmanagement_project_delete",} NaN
Issue Analytics
- State:
- Created 3 years ago
- Reactions:5
- Comments:5 (1 by maintainers)
Top Results From Across the Web
kafka metrics missing in micrometer since SpringBoot 2.3.x
I am not using spring-kafka, just manually creating KafkaStreams or KafkaConsumer, but that worked with 2.2.x. Adding the spring-kafka ...
Read more >Apache Kafka Reference Guide - Quarkus
The Kafka broker preserves order of records inside a partition and not inside a topic. Therefore, it is important to think about how...
Read more >KIP-714: Client metrics and observability
Troubleshooting Kafka problems is currently an organisationally complex issue, with different teams or even organisations running the client applications ...
Read more >Spring for Apache Kafka
If it is not available, the message converter has no header ... By default, a bean with name kafkaListenerContainerFactory is expected.
Read more >All About Kafka, Monitoring and Its Performance Optimization
What is Topic, Partition, Broker, Producer and Consumer in Kafka? ... (but no reference id); Each topic identified by name (Unique Key).
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@shakuzen Thx for the update. I tried version 1.6.0 and the topic tag seems to be available again. Thx a lot 🙂
It looks like the logic to replace meters with less tags had some issues, which hopefully have been addressed by #2312. Please try out version 1.5.6 to see if it is fixed for you.