Bug in new KafkaMetrics (1.4.0+) when using multiple clients and common tags

I was attempting to use the new Kafka meter binders introduced in 1.4.0 and ran into a problem. The application has multiple Kafka producers and consumers, and uses a MeterFilter to apply common tags to all meters. The behavior I’m seeing is that meters only get registered for one consumer and one producer.

I traced the problem to checkAndBindMetrics, specifically where the code removes/ignores meters with “less tags”. The first Kafka client works fine, but all meters for any subsequent client will not be registered because they won’t yet have the common tags applied by the MeterFilter, thus will have “less tags”.

When I comment out this code, all client meters are registered as expected:

                        //Kafka has metrics with lower number of tags (e.g. with/without topic or partition tag)
                        //Remove meters with lower number of tags
//                        boolean hasLessTags = false;
//                        for (Meter other : registry.find(meterName).meters()) {
//                            List<Tag> tags = other.getId().getTags();
//                            if (tags.size() < meterTags.size()) registry.remove(other);
//                            // Check if already exists
//                            else if (tags.size() == meterTags.size())
//                                if (tags.equals(meterTags)) return;
//                                else break;
//                            else hasLessTags = true;
//                        }
//                        if (hasLessTags) return;

I’m sure this code was added for good reason, I’m just not clear on why meters with less tags need to be removed, so I’m hesitant to submit this in a PR.

/cc @jeqo for his thoughts.

Slight tangent: in the course of this investigation I read through several issues & PRs and just wanted to say that I recognize that Kafka client metrics are complicated, and that I appreciate the effort being put forth in this project to try to get it as right as possible – adding support for producer and streams clients, and getting away from JMX. I’m also following PR #1173 re: Kafka client binders using a Kafka metric reporter.

[snagafritz]

Confirmed this has addressed the original issue. Thank you @shakuzen and @jeqo !

[shakuzen]

Thank you for trying out the new Kafka metrics implementation and sorry you’re having an issue with it.

The reason for avoiding registering meters with less tags is because the Kafka client will return metrics e.g. per partition and overall. This does not play well with PrometheusMeterRegistry due to having the same name but a different set of tags. If we register the fine-grained meters and not the summary one, we do not lose anything as a summary can be made by aggregating the more fine-grained meters. Hence we opt to register the more fine-grained meters (those with more tags, e.g. partition).