Memory leak on disconnect() method usage with producer

Hi,

In one of our application we are using nodejs based microservice which internally uses node-rdkafka module to send messages using kafka producer which involves making runtime call scheduled at specific interval to perform below operations:

• Invoking kafka producer connect method
• Calling producer using kafka produce method to send messages
• Invoking kafka producer disconnect method

Disconnect method usage:

let prod =  new Kafka.Producer({
// config properties
})
prod.disconnect()

However, we have been noticing memory leak when calling node-rdfafka producer and suspecting that it is disconnect method which is causing the leak.

Thanks!

[iradul]

I’ve pushed preliminary fix on fix-producer-leaks branch. I still need help with testing. It would be great if some of you can try it out and see if it helps to prevent Producer memory leaks and also check if events are still flowing in as expected.

Also, it seems that Consumer has leaks when rebalance_cb or offset_commit_cb are set and these need to be addressed as well.

[ArneSchulze]

Hi @iradul, I just tried out your fix and it looks like the memory leak in HighLevelProducer is gone now! The memory allocation timeline of our service running with your fix shows a much cleaner heap.

In comparison, this is the memory allocation timeline of our service running on the latest master.

The active objects allocated in the middle of the master graph are HighLevelProducer instances which should have already been garbage collected by the end of the recording