[Doc] MIrrorMaker2 docs don't describe how to enable compression in the target cluster
See original GitHub issueSuggestion / Problem When trying to configure mirrormaker2 I ran into numerous issues trying to get my messages to be compressed in the target cluster. It turns out that mirrormaker2 hands it’s messages off to kafka connect. So any producer settings you make in the sourceConnector have no affect (other than increasing cpu usage due to decompression). To properly enable compression on the destination cluster you need to make the producer setting to the kafkaconnect pieces, under clusters. Like so:
clusters:
- alias: "sourceCluster"
bootstrapServers: my-source-cluster-bootstrap:9094
config:
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
- alias: "targetCluster"
bootstrapServers: my-target-cluster-bootstrap:9092
config:
producer.compression.type: zstd
By contrast consumer specific configuration DOES go in the sourceConnector: mirrors:
- sourceCluster: “sourceCluster” targetCluster: "targetCluster sourceConnector: config: sync.topic.acls.enabled: “false” consumer.fetch.min.bytes: 262144 consumer.fetch.max.wait.ms: 10000 consumer.max.poll.records: 5000 consumer.receive.buffer.bytes: -1 consumer.send.buffer.bytes: -1
Documentation Link https://strimzi.io/docs/operators/master/using.html#proc-mirrormaker-replication-str I’d like to see this as part of the examples in this section.
Also… it would be good to clarify that kafkamirrormaker2 will NOT work if not colocated with the target cluster. (Pull not push, mm2 has no push support).
Issue Analytics
- State:
- Created 3 years ago
- Reactions:3
- Comments:5 (2 by maintainers)
Top GitHub Comments
@Koushikk-24 I think you can work on this if you want
This issue is available to resolve