Memory leak with kafka metrics

Hi there,

I have a springboot app wich purpose is to print the lag on all kafka topics/consumer groups. And that’s all.

It runs out of memory, and the memory usage is as follows :

Code :

@Service
@ManagedResource
public class KafkaLagWriter {

    private static final Logger LOGGER = LoggerFactory.getLogger(KafkaLagWriter.class);

    private final KafkaAdmin kafkaAdmin;
    private final String bootstrapServers;

    public KafkaLagWriter(final KafkaAdmin kafkaAdmin,
                          @Value("${spring.kafka.bootstrap-servers}") final String bootstrapServers) {
        this.kafkaAdmin = kafkaAdmin;
        this.bootstrapServers = bootstrapServers;
    }

    @ManagedOperation
    @Scheduled(fixedDelay = 60_000)
    public void logKafkaTopicLags() {

        try (final AdminClient adminClient = AdminClient.create(kafkaAdmin.getConfig())) {
            final ListConsumerGroupsResult consumerGroupsResult = adminClient.listConsumerGroups();
            final Collection<ConsumerGroupListing> consumerGroupListing = consumerGroupsResult.all().get();
            for (final ConsumerGroupListing consumerGroup : consumerGroupListing) {
                final String consumerGroupId = consumerGroup.groupId();

                final ListConsumerGroupOffsetsResult consumerGroupOffsetsResult = adminClient.listConsumerGroupOffsets(consumerGroup.groupId());
                final KafkaFuture<Map<TopicPartition, OffsetAndMetadata>> futureMap = consumerGroupOffsetsResult.partitionsToOffsetAndMetadata();
                final Map<TopicPartition, OffsetAndMetadata> offSetByPartition = futureMap.get();
                final Map<String, Map<TopicPartition, OffsetAndMetadata>> topicsToOffSetByPartition = new HashMap<>();

                offSetByPartition.forEach((key, value) -> {
                    Map<TopicPartition, OffsetAndMetadata> offSets = topicsToOffSetByPartition.get(key.topic());
                    if (offSets != null) {
                        offSets.put(key, value);
                    } else {
                        offSets = new HashMap<>();
                        offSets.put(key, value);
                    }
                    topicsToOffSetByPartition.put(key.topic(), offSets);
                });

                for (final Entry<String, Map<TopicPartition, OffsetAndMetadata>> topicToPartitions : topicsToOffSetByPartition.entrySet()) {
                    logOffSets(consumerGroupId, topicToPartitions.getKey(), topicToPartitions.getValue());
                }
            }
        } catch (final Exception e) {
            final ExceptionEvent event = ExceptionEventBuilder.createBuilder()
                                                              .method(Events.KAFKA_MONITORING)
                                                              .exceptionClassName(e.getClass().getName())
                                                              .exceptionMessage(e.getMessage())
                                                              .stacktrace(ExceptionUtils.getStackTrace(e))
                                                              .build();
            LOGGER.error(event, null);
        }
    }

    private void logOffSets(final String groupId, final String topic, final Map<TopicPartition, OffsetAndMetadata> offSetByPartition) {
        try (final KafkaConsumer<?, ?> consumer = createNewConsumer(groupId)) {
            final Map<TopicPartition, Long> endOffSetByPartition = consumer.endOffsets(offSetByPartition.keySet());
            for (final TopicPartition topicPartition : offSetByPartition.keySet()) {
                final Long endOffSet = endOffSetByPartition.get(topicPartition);
                final long currentOffSet = offSetByPartition.get(topicPartition).offset();
                final Long lag = endOffSet != null ? endOffSet - currentOffSet : null;

                final KafkaConsumerStatisticEvent event = KafkaConsumerStatisticEventBuilder.createBuilder()
                                                                                            .topic(topic)
                                                                                            .consumerGroup(groupId)
                                                                                            .partition(topicPartition.partition())
                                                                                            .currentOffSet(currentOffSet)
                                                                                            .endOffSet(endOffSet)
                                                                                            .lag(lag)
                                                                                            .build();
                LOGGER.info(event, null);
            }
        }
    }

    private KafkaConsumer<?, ?> createNewConsumer(final String groupId) {
        final Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new KafkaConsumer<>(properties);
    }
}

Can you provide some help on this matter ?

<micrometer.version>1.1.4</micrometer.version>

[izeye]

i dont need these metrics

@gstephant If you don’t need them, you can disable them with the following application property:

spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.KafkaMetricsAutoConfiguration

[alevohin]

@jorgheymans @shakuzen I suppose the problem with memory leak is described here #2096. Version 1.3.X is still used by Springboot 2.2.6.RELEASE