move KafkaConnect and MirrorMaker 1.0/2.0 to statefulsets
See original GitHub issueStatefulset allows having predictable POD names. That, in turn, allows using pod name as group.instance.id
and enable static membership that was designed for cloud applications.
Issue Analytics
- State:
- Created 2 years ago
- Reactions:2
- Comments:10 (8 by maintainers)
Top Results From Across the Web
Migrate from MirrorMaker to Replicator
This topic provides examples of how to migrate from an existing datacenter that is using MirrorMaker to Replicator.
Read more >Migrating clusters using Apache Kafka's MirrorMaker
Makes use of the Apache Kafka Connect framework and ecosystem. Detects new topics and partitions. Automatically syncs topic configuration between clusters.
Read more >Chapter 2. Deployment configuration Red Hat AMQ 2021.q2
Kafka Connect clusters with Source2Image support; Kafka MirrorMaker; Kafka Bridge; Cruise Control. Note. Labels applied to a custom resource are also applied to ......
Read more >How to migrate consumer offsets using MirrorMaker 2.0?
With Kafka 2.7.0, I am using MirroMaker 2.0 as a Kafka-connect connector to replicate all the topics from the primary Kafka cluster to...
Read more >MirrorMaker 2 on Kafka Connect - MSK Labs
Setup Kafka Connect cluster required for MirrorMaker2. Configure and start MirrorMaker2 connectors. (Optional) Start MM2 Consumer Group Offset Sync ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I do not think this has anything to do with this issue. The coordinator simply changes while the Kafka broker restarts. If you have something more, please start a discussion or something and we can continue there: https://github.com/strimzi/strimzi-kafka-operator/discussions
Apologies for the lack of clarity, the issue I am observing is quite similar and it has to do with KafkaConnect. As you can see from the error below, the Group coordinator is fixed hence the consumer cannot locate it after a cluster upgrade involving nodes undergoing rolling upgrade. I am using Strimzi 0.30.0 with Kafka version 3.2.0 hence why I mentioned StrimziPodSet which is now default. This issue is preventing my Connector tasks from executing at polling intervals.
"2022-10-11 09:28:28,738 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Discovered group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,675 INFO [Consumer clientId=consumer-prod-kafka-2, groupId=prod-kafka] Discovered group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-status]" "2022-10-11 09:28:28,637 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Discovered group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,637 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Group coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,637 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Requesting disconnect from last known coordinator prod-kafka-kafka-1.prod-kafka-kafka-brokers.kafka.svc:9093 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]" "2022-10-11 09:28:28,636 INFO [Consumer clientId=consumer-prod-kafka-1, groupId=prod-kafka] Node 2147483646 disconnected. (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread - prod-kafka-connect-offsets]"