Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

KafkaClient partitions_for_topic returns None even when initialized with topic

See original GitHub issue

This is the code snippet that I have…

        self.kafka_client = KafkaConsumer(self.topic, **kafka_config)
        stream_shards = self.kafka_client.partitions_for_topic(self.topic)

I am expecting stream_shards to contain the partitions for the topic. But it is None.

however, if I call self.kafka_client.topics() and then call partitions_for_topic it returns correct partitions.

Issue Analytics

State:
Created 4 years ago
Comments:6 (1 by maintainers)

Top GitHub Comments

1reaction

brouberolcommented, Aug 10, 2019

Indeed, it appears 1f73287 fixed it. Would you consider releasing a patch version to PyPI? The issue itself is fixed but as long as the patch isn’t released, users will still be affected.

Thanks!

1reaction

georgepsarakiscommented, Aug 5, 2019

KafkaConsumer.partitions_for_topic seem as a read-only method, just retrieves data from the protected variable:

https://github.com/dpkp/kafka-python/blob/ea35fdfe1d66eb481e3406ad161a1255573dd50f/kafka/cluster.py#L106

In order to populate the topics and partitions metadata, .update_metadata needs to run:

https://github.com/dpkp/kafka-python/blob/ea35fdfe1d66eb481e3406ad161a1255573dd50f/kafka/cluster.py#L229

which is called by:

https://github.com/dpkp/kafka-python/blob/79dd508b14fd2d66a8b6d32353e8e64989c4ff84/kafka/client_async.py#L807

and finally:

https://github.com/dpkp/kafka-python/blob/79dd508b14fd2d66a8b6d32353e8e64989c4ff84/kafka/client_async.py#L579

So as far as I understand, a single call to to the server is required in order to populate the internal metadata cache, which is used by partitions_for_topic.

I think the solution could be to modify partitions_for_topic to also make a call to .topics if ClusterMetadata._partitions is empty.

Top Results From Across the Web

KafkaConsumer — kafka-python 2.0.2-dev documentation

If topics were subscribed using subscribe() , then this will give the set of topic partitions currently assigned to the consumer (which may...

Kafka-python get number of partitions for topic - Stack Overflow

Might be a slightly simplistic solution, but: from kafka import KafkaClient client = KafkaClient('SERVER:PORT') topic_partition_ids ...

kafka-python Documentation - Read the Docs

A Kafka client that publishes records to the Kafka cluster. ... Return set of all partitions for topic (whether available or not).

org.apache.kafka.common.Cluster java code examples

partitionsForTopic (topic)) { PartitionInfo currPartInfo = curr.partition(new ... public void clearController() { if (cluster.controller() != null) ...

Documentation - Apache Kafka

Kafka only provides a total order over records within a partition, not between different partitions in a topic. Per-partition ordering combined with the ......