consumer.on('event.log', ...) spams "Handle is terminating: failed 0 request(s) in retry+outbuf" upon consumer.disconnect

Thanks for providing a great nodejs library for Kafka!

Environment Information

• OS [e.g. Mac, Arch, Windows 10]: Linux, through Docker on Kubernetes
• Node Version [e.g. 8.2.1]: 10.2.1
• NPM Version [e.g. 5.4.2]: 6.4
• C++ Toolchain [e.g. Visual Studio, llvm, g++]: g++ (through your npm scripts)
• node-rdkafka version [e.g. 2.3.3]: 2.3.3

Steps to Reproduce

1. Connect a new KafkaConsumer (https://github.com/Yolean/kafka-cache/blob/master/lib/kafka.js#L158)
2. Listen for librdkafka messages using conumer.on('event.log', ....) (https://github.com/Yolean/kafka-cache/blob/master/lib/kafka.js#L173)
3. Disconnect the consumer after its usage (https://github.com/Yolean/kafka-cache/blob/master/lib/kafka.js#L201)

Logs get spammed with:

kafka-1.broker.kafka.svc.cluster.local:9092/1: Handle is terminating: failed 0 request(s) in retry+outbuf

Spammed meaning a few hundred messages per second, enough to take a significant amount of CPU, blocking normal operations for the process.

This occurs seemingly at random and is therefore hard to reproduce. But I was hoping to get some insights in what I might be doing wrong here.

node-rdkafka Configuration Settings debug parameter set to broker.

Additional context

[webmakersteve]

I do have a 3.x branch but it doesn’t have those proposed changes just yet. Not much parallel development happens on this project since it’s mostly me over here on this end, and of course all my wonderful contributors helping out otherwise 😄. But I’ll take the suggestion to heart and try to get that going sooner than I originally planned to so the community can see the proposals!

Edit: Turns out that 3.x branch is only local. Will clean it up a bit and push it up.

[solsson]

I am likely going to get rid of it as a C++ land function when a higher order consumer is made in the future because of how many problems it causes and how hard it is to debug.

@webmakersteve That’s very interesting roadmap information. I think it could benefit both new users and this project’s community to have only one way of doing things.

I’m not sure I understood the C++ land precondition, but how about launching a 3.x branch with APIs removed already? It would make the forward path clear and let us evaluate and help document how it affects use cases.