[Question] How to attach MirrorMaker to and existing group.id in source?

To migrate from MM1 to MM2 I would like to start MM2 read from where MM1 stopped. I tried to attach to the same consume group as MM1 read from. Here is my configuration:

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  name: strimzi-aws-to-dc
spec:
  replicas: 1
  connectCluster: "dc"
  clusters:
  - alias: "dc"
    bootstrapServers: dc:9092
  - alias: "aws"
    bootstrapServers: aws:9092
    config:
      group.id: aws-to-dc-stg-group
  mirrors:
  - targetCluster: "dc"
    sourceCluster: "aws"
    sourceConnector:
      config:
        group.id: aws-to-dc-stg-group
    topicsPattern: "mm1-switch-mm2"
    groupsPattern: "aws-to-dc-stg-group"

When I run kubectl describe deployments I see that Kafka connect uses the default group:

...
    Environment:
      KAFKA_CONNECT_CONFIGURATION:      config.storage.topic=mirrormaker2-cluster-configs
                                        group.id=mirrormaker2-cluster
                                        status.storage.topic=mirrormaker2-cluster-status
                                        config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
                                        offset.storage.topic=mirrormaker2-cluster-offsets
                                        config.providers=file
                                        value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
                                        key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
...

In MM2 logs I also see that it does not use groupId what expected:

...
2020-06-26 15:16:55,940 INFO ConsumerConfig values:
...
	group.id = null
	group.instance.id = null
...

2020-06-26 15:16:56,076 INFO [Consumer clientId=consumer-4, groupId=null] Subscribed to partition(s): mm1-switch-mm2-0 (org.apache.kafka.clients.consumer.KafkaConsumer) [task-thread-aws->dc.MirrorSourceConnector-0]
2020-06-26 15:16:56,077 INFO Starting with 0 previously uncommitted partitions. (org.apache.kafka.connect.mirror.MirrorSourceTask) [task-thread-aws->dc.MirrorSourceConnector-0]
2020-06-26 15:16:56,078 INFO [Consumer clientId=consumer-4, groupId=null] Seeking to offset 10 for partition mm1-switch-mm2-0 (org.apache.kafka.clients.consumer.KafkaConsumer) [task-thread-aws->dc.MirrorSourceConnector-0]
2020-06-26 15:16:56,078 INFO task-thread-aws->dc.MirrorSourceConnector-0 replicating 1 topic-partitions aws->dc: [mm1-switch-mm2-0]. (org.apache.kafka.connect.mirror.MirrorSourceTask) [task-thread-aws->dc.MirrorSourceConnector-0]
2020-06-26 15:16:56,078 INFO WorkerSourceTask{id=aws->dc.MirrorSourceConnector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-aws->dc.MirrorSourceConnector-0]
2020-06-26 15:16:56,085 INFO [Consumer clientId=consumer-4, groupId=null] Cluster ID: olRs7ybdTU--T9NXZvIbSw (org.apache.kafka.clients.Metadata) [task-thread-aws->dc.MirrorSourceConnector-0]

Please advise how to attach to the consumer group.

[yupyvovarov]

Already found the answer. I missed is that Kafka is a key/value system. So the solution is quite simple:

1. Create mirrormaker2-cluster-offsets topic in destination side: $ ./kafka-topics.sh --bootstrap-server kafka-dc:9092 --create --replication-factor 1 --partitions 1 --topic mirrormaker2-cluster-offsets --config "cleanup.policy=compact"
2. Populate topic with requared offset: $ ./kafka-console-producer.sh --bootstrap-server kafka-dc:9092 --topic mirrormaker2-cluster-offsets --property parse.key=true --property key.separator="|" ["aws->dc.MirrorSourceConnector",{"cluster":"aws","partition":0,"topic":"mm1-switch-mm2"}]|{"offset":21}
3. Start MM2.

MM2 will start replication from the defined offset.

[scholzj]

Thanks for updating the case with the solution! That is always great to have it here for record. 👍