bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
See original GitHub issueDescribe the bug
The prometheus metric bookie_journal_JOURNAL_QUEUE_SIZE
exposed by bookie is negative. Shouldn’t queue lengths be positive
To Reproduce
- Launch Pulsar on Kubernetes
- Inspect metrics endpoint of Bookkeeper
$ curl http://platform-pulsar-bookkeeper:8000/metrics | grep QUEUE_SIZE
# TYPE bookie_journal_JOURNAL_CB_QUEUE_SIZE counter
bookie_journal_JOURNAL_CB_QUEUE_SIZE 57253790
# TYPE bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE counter
bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE 0
# TYPE bookie_journal_JOURNAL_QUEUE_SIZE counter
bookie_journal_JOURNAL_QUEUE_SIZE -57253790
Expected behavior Queue lengths are zero or above
Screenshots
Additional context
- Pulsar
2.6.1
- Kafkaesque Helm Chart
1.0.24
- Probably has not bearing but
exposeTopicLevelMetricsInPrometheus
is set tofalse
as we have a lot of topics
Issue Analytics
- State:
- Created 3 years ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
No results found
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
That’s a helpful observation. When I was looking through the code last night, I was wondering if there was any type of relationship between those two metrics.
@michaeljmarshall Thanks so much for hunting this down. Closing this ticket . . . .