Better way to check if connection with Kafka is working

First, thanks for the awesome module, it’s always helping in my projects. But I have an use case where I need to periodically check if the connection and the Kafka cluster is alive. Right now, to do this I created an auxiliary topic and that I will write a message and then try to read it to see if the Kafka cluster is running and if communication with the cluster is possible.

The problem, is that this solution is kind cumbersome, for instance I need to create and maintain an auxiliary topic, and if I’m only using KafkaProducer I need to create an extra KafkaConsumer just to check this, the same happens in the inverse scenario.

So would be possible to improve this use case?

Thanks

[andrewkowalik]

This pattern seems to go against the design of Kafka I think? Producers and consumers should be decoupled.

The producer should only care about message delivery. I believe the library already takes care of this.

• Producers raise exceptions when connection to the cluster fails.
• The Producer client has a configurable acks keyword argument that sets the level of acknowledgment to None, Leader or ISR (in-sync replicas). I would think leader acknowledgement should get you a good enough guarantee that the producer has succeeded. Of course this fails if the leader dies before a follower has copied it.

The consumer only cares about consuming messages which I also think the library handles in regards to connection status.

It could be beneficial to run a canary service if thats what you really care about. We have done something similar in the past where we were curious about message publishing/delivery latency and wrote a simple canary service that both publishes and consumes its own messages. You could do something similar if you just want to monitor kafka but keep in mind that you don’t need to do this for every producer/consumer, just a single instance could be doing this.

[andrewkowalik]

You could build a canary service for something like that, not sure it meets the requirements for this library but its also only a few lines of code. Would not be that hard to whip up. I will document how I have done it before.

---

Drew Kowalik

On Thu, Jul 26, 2018 at 11:39 AM, Maulik Shah [e-mail] wrote:

@infecto https://github.com/infecto I have to monitor kafka, and my current task is to get something like “is kafka up and online?”

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dpkp/kafka-python/issues/1137#issuecomment-408194971, or mute the thread https://github.com/notifications/unsubscribe-auth/ACZE9IPeRAnI-TIUYfUq3vD_0PuZY59gks5uKgzwgaJpZM4OMvAV .