Urgent - Worker restart , always have a number of duplicate message in topic
See original GitHub issueHi everyone , I’m implementing uReplicator with latest master branch ( commit 70ac3404bb4517756c6813d3962b77edc7700019 ). As the another commit on master , after Worker restart, always have lots of message duplicate in Destination topic .
Here is overview my step :
step1: Build the latest code
step2: build Docker image with this Dockerfile
FROM openjdk:8-jre
COPY confd-0.15.0-linux-amd64 /usr/local/bin/confd
COPY uReplicator/uReplicator-Distribution/target/uReplicator-Distribution-pkg /uReplicator
COPY uReplicator/config uReplicator/config
COPY confd-new /etc/confd
#I use confd to generate configuration file in Container
COPY entrypoint-new.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh && \
chmod +x /usr/local/bin/confd && \
chmod +x /uReplicator/bin/*.sh &&
ENTRYPOINT [ "/entrypoint.sh" ]
Step3: prepare 2 topic:
- Source topic : 500 message & keep rising to target 1000 message.
- Destination topic : Blank topic - 0 message
step4: apply deployment on Kubernetes with config :
- Controller : 1 pod : resource require: 100m CPU - 1500Mi memory
- Worker : 2 pod : resource require: 100m CPU - 1500Mi memory
clusters.properties
kafka.cluster.zkStr.cluster1=src-zk:2181
kafka.cluster.servers.cluster1=src-kafka:9092
kafka.cluster.zkStr.cluster2=dst-zk:2181
kafka.cluster.servers.cluster2=dst-kafka:9092
consumer.properties
zookeeper.connect=src-zk:2181
bootstrap.servers=src-kafka:9092
zookeeper.connection.timeout.ms=30000
zookeeper.session.timeout.ms=30000
group.id=kloak-mirrormaker-test
consumer.id=kloakmms01-sjc1
socket.receive.buffer.bytes=1048576
fetch.message.max.bytes=8388608
queued.max.message.chunks=5
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
auto.offset.reset=smallest
dstzk.properties
enable=true
zkServer=src-zk:2181
helix.properties : disable instanceId because of i’m using more than 1 Worker
zkServer=src-zk:2181
#instanceId=testHelixMirrorMaker01
helixClusterName=testMirrorMaker
federated.deployment.name=uReplicator-Tests
producer.properties
bootstrap.servers=dst-kafka:9092
client.id=kloak-mirrormaker-test
Step4: check number of message on both server
Src & Dst topic : 600 message
Step5 : Delete 1 worker POD kubernetes by : kubectl delete pod … ( no --force command )
result : 1 - old POD is Terminating. Src & Dst keep sync up on time : 750 -750 on both topic
2 - a few second later: new POD is UP Src topic : keep rising to 900 -> 95- -> 1000 Dst topic : stay at 750 message
3 - a minutes later : Src topic : 1000 message Dst topic : more than 1000 message : sometime 1200 message , sometime 1100-1300…etc ( the result is different in every test )
Note : if no Worker is down. The replication result is perfect . But in this case in everytime is test , the result is different and uReplicator is not stable. @yangy0000 @Pawka @DAddYE @hparra @chirayuk @dentafrice CC to anyone Who contributed to this application . Please let me know/correct me if something went wrong .
Issue Analytics
- State:
- Created 4 years ago
- Comments:21 (10 by maintainers)
Top GitHub Comments
yes ,perfect .
@Technoboy- hey bro , the problem is : kill process or kill -9 process . I did this way with
kill -9
I’ve tried with ‘kill processId’ and here is result : have commit offset log? And It’s Work !!!