Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

zookeeper logs show QuorumPeer Error closing file

See original GitHub issue

Deployed Pravega-Operator image: pravega/pravega-operator:0.3.2 Pravega build used : 0.5.0-2189.cabe11b Zookeeper-operator image: pravega/zookeeper-operator:0.2.1 Ran small i/o workload (Total - 4 readers, 3 writers, 120 events/sec, ~ 100 KB/sec IO)

Environment details: PKS / K8 with medium cluster:

3 master nodes @ large.cpu (4 CPU, 4 GB Ram, 16 GB Disk)
10 worker nodes @ xlarge.cpu(8 cpu, 8 GB Ram, 32 GB Disk)
Tier-1 storage is from VSAN datastore
Tier-2 storage curved on NFS Client provisioner using Isilon as backend

See these errors in zk-0 , zk-1 and zk-2 even without restarting any zookeepers.

2019-04-04 09:59:37,732 [myid:1] - ERROR [ProcessThread(sid:1 cport:-1)::QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-04-04 09:59:37,736 [myid:1] - ERROR [LearnerHandler-/172.24.2.3:50172:QuorumPeer@1596] - Error closing file:
2019-04-04 09:59:37,737 [myid:1] - WARN  [LearnerHandler-/172.24.2.3:50172:QuorumPeer@1527] - Restarting Leader Election
2019-04-04 09:59:37,737 [myid:1] - INFO  [pravega-zk-0.pravega-zk-headless.default.svc.cluster.local/172.24.2.2:3888:QuorumCnxManager$Listener@892] - Leaving listener

zk_logs.zip

Issue Analytics

State:
Created 4 years ago
Comments:17 (8 by maintainers)

Top GitHub Comments

1reaction

Raghuveera-Raocommented, Jul 2, 2019

Yes… all pvc were deleted… $ kubectl get pvc

NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cache-pravega-pravega-segmentstore-0    Bound    pvc-603b878f-9ccd-11e9-aede-005056ba320d   30Gi       RWO            standard       26m
cache-pravega-pravega-segmentstore-1    Bound    pvc-88daf068-9ccd-11e9-aede-005056ba320d   30Gi       RWO            standard       25m
cache-pravega-pravega-segmentstore-2    Bound    pvc-9cd13017-9ccd-11e9-aede-005056ba320d   30Gi       RWO            standard       24m
data-pravega-zk-0                       Bound    pvc-e649cf07-9ccc-11e9-aede-005056ba320d   20Gi       RWO            standard       29m
data-pravega-zk-1                       Bound    pvc-00bb9f5f-9ccd-11e9-aede-005056ba320d   20Gi       RWO            standard       29m
data-pravega-zk-2                       Bound    pvc-1686f582-9ccd-11e9-aede-005056ba320d   20Gi       RWO            standard       28m
index-pravega-bookie-0                  Bound    pvc-60006dc1-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
index-pravega-bookie-1                  Bound    pvc-60096fd1-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
index-pravega-bookie-2                  Bound    pvc-601c4e9d-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
index-pravega-bookie-3                  Bound    pvc-602b82f0-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
journal-pravega-bookie-0                Bound    pvc-5ffc7aed-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
journal-pravega-bookie-1                Bound    pvc-600dd9b4-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
journal-pravega-bookie-2                Bound    pvc-6013e6a4-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
journal-pravega-bookie-3                Bound    pvc-602e30fe-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-0                 Bound    pvc-5ffe34d0-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-1                 Bound    pvc-600563c9-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-2                 Bound    pvc-60178a4e-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-3                 Bound    pvc-60284144-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
pravega-tier2                           Bound    pvc-575fabf9-9ccd-11e9-aede-005056ba320d   2000Gi     RWX            nfs            26m
pvc-shaka-zulu-nfs-client-provisioner   Bound    pv-shaka-zulu-nfs-client-provisioner       10Mi       RWO                           27m

0reactions

Raghuveera-Raocommented, Jul 3, 2019

@pbelgundi @Raghuveera-Rao I don’t see where this conversation is going. It started with an error to write to a file, and now it became about an error message that out of context is meaningless. If there is no real issue being discussed, please close it.

Will close this if '‘ERROR [QuorumPeer’ error comes with this …

Top Results From Across the Web

Zookeeper connection error - Stack Overflow

So Zookeeper is configured to log under /var/log. Then there's usually a zookeeper.log and/or zookeeper.out file which should indicate your startup error.

ZooKeeper Administrator's Guide - Apache ZooKeeper

The Data Directory; The Log Directory; File Management ... meaning that it will shutdown (process exit) if an error occurs that it cannot...

Read timeout errors and exceptions are seen on all three ...

There are timeout errors and exceptions seen on all three controller zookeeper logs. From /var/log/cloudnet/cloudnet_java-zookeeper.

Solved: Unable to start the zookeper server in HDP 2.6

I am Facing the issue with zookeeper. while starting the service from ambari console log files are not updating. in .out log file....

UCS Hyperflex Zookeeper Service Explanation and Verification

Query Zookeeper Service. Zookeeper Log Files in a Live Setup. Zookeeper Log Files from Support-bundle (storfs) ...