question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Doc] MIrrorMaker2 docs don't describe how to enable compression in the target cluster

See original GitHub issue

Suggestion / Problem When trying to configure mirrormaker2 I ran into numerous issues trying to get my messages to be compressed in the target cluster. It turns out that mirrormaker2 hands it’s messages off to kafka connect. So any producer settings you make in the sourceConnector have no affect (other than increasing cpu usage due to decompression). To properly enable compression on the destination cluster you need to make the producer setting to the kafkaconnect pieces, under clusters. Like so:

  clusters:
  - alias: "sourceCluster"
    bootstrapServers: my-source-cluster-bootstrap:9094
    config:
      config.storage.replication.factor: 1
      offset.storage.replication.factor: 1
      status.storage.replication.factor: 1
  - alias: "targetCluster"
    bootstrapServers: my-target-cluster-bootstrap:9092
    config:
      producer.compression.type: zstd

By contrast consumer specific configuration DOES go in the sourceConnector: mirrors:

  • sourceCluster: “sourceCluster” targetCluster: "targetCluster sourceConnector: config: sync.topic.acls.enabled: “false” consumer.fetch.min.bytes: 262144 consumer.fetch.max.wait.ms: 10000 consumer.max.poll.records: 5000 consumer.receive.buffer.bytes: -1 consumer.send.buffer.bytes: -1

Documentation Link https://strimzi.io/docs/operators/master/using.html#proc-mirrormaker-replication-str I’d like to see this as part of the examples in this section.

Also… it would be good to clarify that kafkamirrormaker2 will NOT work if not colocated with the target cluster. (Pull not push, mm2 has no push support).

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:3
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
scholzjcommented, Jun 10, 2021

@Koushikk-24 I think you can work on this if you want

0reactions
Koushikk-24commented, Jun 17, 2021

This issue is available to resolve

Read more comments on GitHub >

github_iconTop Results From Across the Web

Mirror Maker 2 doesn't follow compression.type from source
I was able to turn on the GZIP compression to MM2 producers to the Target Cluster (Cluster2). I do not know if this...
Read more >
Kafka mirroring (MirrorMaker) - Apache Software Foundation
The following diagram shows how to use the MirrorMaker tool to mirror a source Kafka cluster into a target (mirror) Kafka cluster. The...
Read more >
MirrorMaker 2.0 (MM2) - Amazon MSK Migration Guide
You can use a single MM2 cluster to migrate data between multiple clusters. MM2 automatically detects new topics and partitions, while also ensuring...
Read more >
Tutorial: Using Cluster Linking for Topic Data Sharing
Tutorial on a basic Cluster Linking use case to mirror topic data across clusters. ... You are currently viewing Confluent Platform documentation.
Read more >
A look inside Kafka Mirrormaker 2 - Cloudera Blog
In settings where there are multiple clusters across multiple data ... MM2 filters out any topics that carry the target cluster name in...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found