Consumer has 500ms delay

The problem make me annoying these days. We used consumer to consume data from kafka, and found its had 500ms delay at fixed-frequency. We checked all the configuration of librdkafka, and found nothing can reduce the delay. And then I add the event listener of event.log, and print the partition and offset of current received message to console. Then I found that the fetch event was almost 500ms before the consumer’s data on same partition and offset. There must be something prevent the consumer from returning the data received at once! So I check the code of node c++ and c. Finally, I found a piece of suspicious code: https://github.com/Blizzard/node-rdkafka/blob/8d2963bf21ed9b8a79ad5adabcec2362bc12392e/src/workers.cc#L450-L459

There is a magic number in the code, and it not indicated on the document, and not supply an option argument for developer to change it.

Sleep function would not make current thread busy, even you call it with 1ms, I think 500ms is not a optimal value as default.

[Tapppi]

Looks like the loop version of consume needs a similar optimisation to https://github.com/Blizzard/node-rdkafka/commit/e82bb2ea93d03e56ed9f1f1a1d2fdfaf22220d00#diff-d3b92fea9df0e70b4f0aed1821f7b6c9 . It is indeed likely that you are producing the message at such a point that the consumer is sleeping at that point. Looking at the code, there are multiple edge cases where that can happen either to the first new message or even during consuming already existing messages.

[yunnysunny]

The pull request https://github.com/Blizzard/node-rdkafka/pull/769 has been merged