Proper delete_topics example required (kafka-python 2.0.2)

Description

Calling the delete_topics function doesn’t seem to work, even when sleeping script for several seconds.

Concretly, could you provide a working example of topic deletion for 2.0.2 ?

How to reproduce

Code snippet :

# Setting up brokers with topic
topic_name = "sentences"
topic_list = [
    NewTopic(
        name=topic_name,
        num_partitions=1,
        replication_factor=1,
        topic_configs={'retention.ms': '3600000'}
    )
]

# Retrieving already-created list of topics
client = KafkaClient(bootstrap_servers=kafka_servers)
metadata = client.cluster
future = client.cluster.request_update()
client.poll(future=future)
broker_topics = metadata.topics()

# Deleting topic if it already exists
# to update configuration (topic_configs)
admin_client = KafkaAdminClient(bootstrap_servers=kafka_servers)
if topic_name in broker_topics:
    deletion = admin_client.delete_topics([topic_name])
    time.sleep(5)
admin_client.create_topics(new_topics=topic_list, validate_only=False)

Error returned :

producer_1  | Traceback (most recent call last):
producer_1  |   File "/usr/app/producer.py", line 42, in <module>
producer_1  |     admin_client.create_topics(new_topics=topic_list, validate_only=False)
producer_1  |   File "/usr/local/lib/python3.7/site-packages/kafka/admin/client.py", line 461, in create_topics
producer_1  |     return self._send_request_to_controller(request)
producer_1  |   File "/usr/local/lib/python3.7/site-packages/kafka/admin/client.py", line 409, in _send_request_to_controller
producer_1  |     .format(request, response))
producer_1  | kafka.errors.TopicAlreadyExistsError: [Error 36] TopicAlreadyExistsError: Request 'CreateTopicsRequest_v3(create_topic_requests=[(topic='sentences', num_partitions=1, replication_factor=1, replica_assignment=[], configs=[(config_key='retention.ms', config_value='3600000')])], timeout=30000, validate_only=False)' failed with response 'CreateTopicsResponse_v3(throttle_time_ms=0, topic_errors=[(topic='sentences', error_code=36, error_message="Topic 'sentences' already exists.")])'

Checklist

• Python package version : 2.0.2
• Apache Kafka broker version: Docker image confluentinc/cp-kafka:5.5.1
• Operating system: Ubuntu 18.04

Thanks for your help

[jeffwidman]

If you take a pcap of the network traffic, I suspect you’ll see that the library issued the delete call, the broker acknowledged and responded with no error, confirming it’s begun the deletion process. Then you’ll see the broker responds to the create call saying the topic already exists… even though the broker just told you it’d started deletion of the topic.

In other words, you could use any library (including Java) and have the exact same problem… as long as the create call hits the broker faster than the broker can complete deletion of the topic.

As I mentioned, I hit the same problem before, and I initially attempted to solve by then querying the broker for an updated list of topics to say “has this topic deletion finished?” And I observed the following sequence against a single-broker cluster:

1. delete topic
2. query for current list of topics… sometimes broker says topic exists, sometimes that it doesn’t (ie, topic deletion is async on the broker side)
3. loop querying broker metadata until the response doesn’t include the deleted topic
4. try to create topic, broker occasionally complains topic exists (this indicates the metadata response is racy with some part of the deletion)
5. sleep 100 milliseconds to give broker time to finish the last little bit of topic deletion
6. try to create topic, broker says topic created successfully

So again, this isn’t a kafka-python library issue, it’s that the broker response is racy from the time you trigger topic deletion until the topic is actually deleted… and so this situation will be true of any library. If you aren’t observing it in other libraries, most probably it’s because the broker is deleting the topic too fast to trigger the race condition.

If you can demonstrate a pcap or kafka-python logs showing that the sequence is different, then I’d be happy to take another look.

[Captain0X]

I have a suggest：

    try:
        dr = producer.delete_topics([producer_params_topic,producer_data_topic],timeout_ms=3000)
        print('删除后的topic列表:',ac.topics_list)
        # print(producer.delete_topics([producer_params_topic]))
        # print(producer.delete_topics([producer_data_topic]))
    except UnknownTopicOrPartitionError:
        print('topic delete fail')
    while True:
        try:
            cr=producer.create_topics({producer_params_topic:slave_num})
            if cr.topic_errors[0][1]==0:
                print(cr)
                producer.create_topics({producer_data_topic:1})
                break
        except Exception as exp:
            print(exp)
        time.sleep(0.5)

as code show, first delete topic ,next step enter a while until it can create a new topic。