Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Question] Kafka Connect status topic is not updated

See original GitHub issue

Hey All,

We are running kafka connect cluster with strimzi. When we deploy a new kafka connector, sometimes connector gets wrong worker id ( wordker ID with the IP which doesn’t exist anymore, because of new deployment), and it is in unassigned state.

We have to manually delete kafka connect statuses topic to be able to refresh the worker id.

How can we tackle this case ?

Any suggestions are appreciated.

Thanks.

Issue Analytics

State:
Created 3 years ago
Comments:11 (11 by maintainers)

Top GitHub Comments

1reaction

klalafaryancommented, Jul 21, 2020

Seems like we have found a problem, and it is related to the config.storage.topic topic which has been created with 3 partitions but it should be created always with single partition.

config.storage.topic The name of the topic where connector and task configuration data are stored. This must be the same for all Workers with the same group.id. Kafka Connect will upon startup attempt to automatically create this topic with a single-partition and compacted cleanup policy to avoid losing data, but it will simply use the topic if it already exists. If you choose to create this topic manually, always create it as a compacted topic with a single partition and a high replication factor (3x or more).

So closing this ticket.

0reactions

klalafaryancommented, Jul 11, 2020

@scholzj Thanks for your reply. We have the same problem with the kafka connect cluster which we have deployed without strimzi. Seems like it is not related to strimzi, so I have created a bug ticket for kafka connect: https://issues.apache.org/jira/browse/KAFKA-10253

Top Results From Across the Web

Kafka Connect : test.sink.txt is not getting updated after ...

1 Answer 1 ... The Kafka file source connect keeps the last read location from file. When you deleted the file without resetting...

kafka connect internal topics - Google Groups

Hi, I have question regarding the internal topics used by Kafka Connect. ... will not create a new worker separate from an existing...

Kafka Connect Concepts | Confluent Documentation

Update the record's topic field as a function of the original topic value and the record's timestamp field. RegexRouter, Not currently available for...

Kafka 3.3 Documentation

Other servers run Kafka Connect to continuously import and export data as ... it can also show you details such as the partition...

Running Kafka Connect - Standalone vs Distributed Mode ...

For resiliency, this means answering the question, “what happens if a particular worker goes offline for any reason?”. Horizontal scale and failover resiliency ......