Memory leak with the consume(n) method and limit in consumeLoop method

Hi,

we have been noticing an issue with how our consumers behave since some time ago and we cannot figure what the issue might be. The only explanation is something in the internals of the library that escapes our understanding and we cannot manage properly. Let me explain:

We have a series of services, each of them defining a consumer and a producer. These services will basically read a topic, used a message pipe, react to them and produce back a message when it has finished. The way we are consuming messages is with the following configuration:

        this.consumerConfig = {
            "group.id": groupName,
            "metadata.broker.list": brokerList,
            "enable.auto.commit": false,
            "enable.auto.offset.store": false,
            "max.poll.interval.ms": maxPollMs
        };
        this.consumerTopicConfig = {
            "enable.auto.commit": false,
            "auto.offset.reset": offsetReset
        };
        this.consumer = new KafkaConsumer(this.consumerConfig, this.consumerTopicConfig);
        // Later in the class
        this.consumer.on('ready', () => {
            this.consumer.subscribe(topicNames);
            this.streamTimeout = setInterval(() => {
                this.consumer.consume(1);
            }, 1000);
        })

What we do later is to react to data events to handle the message. We observed a memory leak since September 9th although this date might not be important since the environment is set in docker and a new release had not been done since August. Without any changes on the consumer itself, or anything related with the consumer, memory usage starts to increase without being cleaned by the garbage collector.

I am not sharing what we do when consume data because this behaviour is found even when leaving the service running with no messages. It does not matter if there are messages to handle or not, the fact that the consumer is consuming makes memory usage go up with no limit. We narrowed it down to this part of the code and if we just call that method once, then no memory leak appears and everything behaves properly.

We found that if we use the consumeLoop call (by calling consume with no arguments) then the memory issue disapears as well. This is

            this.consumer.setDefaultConsumeTimeout(1000);
            this.consumer.consume();

Here we can consume in the same way and memory behaviour is way better. Nonetheless there is an issue with this. When we launch the entire suite that can include up to 9 services (running in the same container) only 4 of them consume data, like if there was a limit to the number of consumers that can consume data in this way.

Each consumer is independent but all of them are running in the same node app. If we split each service in different containers, running 9 node apps each of them with one consumer then the issue disappears and all consumers read properly.

This is our end goal but given the limitations we have now it is not easy for use to switch the infrastructure in the short term and I would like to hear your opinion about this.

Why do you think the memory leaks happens and if is there any limit in the consumers that can run simultaneously with the consume() method. I have not been able to answer any of these questions so far and would love some input about it.

Thanks!

[fuqiang2310]

@AAraKKe I recently encountered the same memory issue with the consume(n) method. After digging into it I found out the way how this method work in librdkafka is, whenever this function is called, it will create a new worker to poll messages from kafka broker. The worker will not die until it polls n number of messages. Therefore in your code:

this.streamTimeout = setInterval(() => {
                this.consumer.consume(1);
            }, 1000);

When there is no new message, you are still creating new workers to poll 1 message every second.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.