MirrorMaker 2 creates topics but doesn't mirror the data

Hi there,

Hope you’re all doing good. In the past I came with questions and got answers and hopeful this time as well.

Let me explain the issue I face! What is the issue? I have setup MirrorMaker2 between two clusters, in the target cluster the topics from the source cluster are created, like (source-cluster.topic-1) but the data are not mirrored.

Some more details of how I ended up here: Initially I had setup the MM2 between two Kafka clusters and everything was working fine until I deleted the target cluster as I had to make some changes to persistent storage for the target cluster. When I recreated the target cluster with the same name, I ended up with the above mentioned issue.

Just to experiment, I also tried creating the target cluster with a different name and now everything worked fine.

So to summarize, when I recreate the target cluster with the same old name, though the topics are created, the data is not mirrored. Any suggestions or possible fixes ?

I don’t have anything much crazy going on from the logs. Just the below one I see but not sure how does this affect.

MirrroMaker Logs: 2020-07-24 10:34:39,251 ERROR Plugin class loader for connector: org.apache.kafka.connect.mirror.MirrorCheckpointConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@57459491 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [pool-3-thread-2] 2020-07-24 10:34:37,060 ERROR Plugin class loader for connector: 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@57459491 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [pool-3-thread-1]

Thanks, Eazhilan

[eazhilan-nagarajan]

Hi @ajborley ,

Thanks for the detailed explanation of how the internals work. I think your suggestion of “removing the spec.mirrors, delete the mm2-offset-syncs.my-target-cluster-1.internal topic and add the spec.mirrors again” trick worked.

Now my target cluster runs with the same old name (my-target-cluster-1) and the KMM2 is able to mirror the data but TBH, I’m not sure if it’s working just because of removing and adding the spec.mirrors alone. Why I think so is, even in my previous attempts I have deleted the mm2-offset-syncs.my-target-cluster-1.internal topic but never touched the spec.mirrors.

Seems like I got an answer this time as well ☺️

Thank you @scholzj and @ajborley for the time and help.

[ajborley]

@eazhilan-nagarajan

This may be due to data in the mm2-offset-syncs.my-target-cluster-1.internal topic on your source cluster that is created and used by the MM2 MirrorSourceConnector. That topic is used to store the offset position that has been reached by the consumer in the MirrorSourceConnector for each mirrored topic-partition. It also stores the mapping of the offset position on the source cluster to the equivalent offset position on the target cluster.

If you did not delete this topic on the source cluster when you recreated the target cluster, the MM2 connectors will start up, read the mm2-offset-syncs.my-target-cluster-1.internal topic and think that you have mirrored records up to the stored offset for each mirrored topic-partition (topic creation is handled separately to record mirroring).

If there is no new data coming into the source topic, then no data will get mirrored, as the MM2 MirrorSourceConnector thinks it has already mirrored the data. If there is new data coming into the topic, the new data should be mirrored, but the historic data will not.

When you created my-target-cluster-4, a new empty mm2-offset-syncs.my-target-cluster-4.internal topic was created on the source cluster, so all the data was mirrored.

I don’t yet understand why data wasn’t mirrored when you created my-target-cluster-1-0, but it may be worth removing the spec.mirrors from your KafkaMirrorMaker2 CR to stop the connectors, deleting the mm2-offset-syncs.my-target-cluster-1-0.internal topic on the source cluster, and then add the spec.mirrors back again, so that the connectors start up again. This should create a new empty mm2-offset-syncs.my-target-cluster-1-0.internal topic on the source cluster, which will mean all the historic data on the source topics should be mirrored by the MM2 MirrorSourceConnector.

Another option is to simply rename the target cluster alias in your KafkaMirrorMaker2 CR spec.clusters list (and copy the new alias name to spec.connectCluster) - this will give MM2 a new alias name for the target cluster, so a new empty mm2-offset-syncs.<target-cluster-alias>.internal topic will be created on the source cluster,