Preserving Partition in the Destination Cluster
See original GitHub issueHi All, This is great work guys in addressing all the short comings of mirror maker . We are trying to see if we can replace this our current setup of mirror with uMirrorMaker before we go live . We have a strict requirement to preserve ordering of messages in destination cluster to be same as the source cluster . I was going through the code and looks like there is a message handler support . Is this the only change we need to do to get it going?
private object defaultMirrorMakerMessageHandler extends MirrorMakerMessageHandler {
override def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): util.List[ProducerRecord[Array[Byte], Array[Byte]]] = {
// rewrite topic between consuming side and producing side
val topic = topicMappings.get(record.topic).getOrElse(record.topic)
//Changed to include source partition information
Collections.singletonList(new ProducerRecord[Array[Byte], Array[Byte]](topic, record.partition(),record.key(), record.message()))
}
}
Also do you guys have any plans on enabling this as a run time parameter so that we can override it.
Issue Analytics
- State:
- Created 7 years ago
- Comments:7 (4 by maintainers)
Top Results From Across the Web
Replicator Overview | Confluent Documentation
Dynamic topic creation in the destination cluster with matching partition counts, replication factors, and topic configuration overrides.
Read more >About SnapMirror SVM replication - Product documentation
The -identity-preserve false option replicates only the volumes and authentication and authorization configurations of the SVM, ...
Read more >13.15. Installation Destination Red Hat Enterprise Linux 7
Keeping the partition selected in the left pane, select the configuration button below the pane to open the Configure Mount Point dialog. Select...
Read more >Managing partitioned tables | BigQuery - Google Cloud
Copy individual partitions · -a or --append_table appends the data from the source partition to an existing table or partition in the destination...
Read more >Documentation - Apache Kafka
The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
If #partition in src is bigger than #partition in dst temporarily, worker will hash the partition into dst. After you expand in dst, it will go back to 1:1 mapping again.
@daluu No. We have merged federated ureplicator into master and current master supports both modes.