Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
zookeeper leader log file grows and fills up disk space
See original GitHub issueIssue
When Bk-client deletes the ledger, it also tries to delete ledger’s parent nodes blindly and because of that ZK’s entire log file fills up with below log Directory not empty for. Broker rollovers cursor’s metadata-ledger at every few configured hours. Broker with large number of global topics and cursors it generates really huge log (>10GB/day) file at zk-leader which consumes most of the disk space.
Tue May 14 00:00:00 2019: 2019-05-14 00:00:00,297 - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x123454567 type:delete cxid:0xc655f6 zxid:0x123454567 txntype:-1 reqpath:n/a Error Path:/ledgers/000/0000/0123/4567 Error:KeeperErrorCode = Directory not empty for /ledgers/000/0000/0123/4567
Possible fix
- Logging configuration change at ZK/Change log level but then it will prevent us to store important logs when it needs for troubleshooting.
- Better solution : BK-client should only delete parent node if it’s empty. It also prevents unexpected data-loss due to unseen zk’s bug while deleting non-empty znode.
Issue Analytics
- State:
- Created 4 years ago
- Reactions:2
- Comments:7 (7 by maintainers)
Top Results From Across the Web
How to reduce the disk space of logs and data for Zookeeper
Clear out current logs: Stop shared services and delete the log files. ... When enabled, ZooKeeper auto purge feature retains the autopurge.
Read more >ZooKeeper Administrator's Guide
Occasionally, when a log grows large, a snapshot of the current state of all znodes will be written to the filesystem and a...
Read more >After upgrading the Control-M/Enterprise Manager to 9.0.20 ...
After upgrading Control-M/Enterprise Manager (EM) to 9.0.20 on Windows, afterward the Kafka log directory is filling up with logA* files which ...
Read more >Disk space issues due to Zookeeper logs size
The Zookeeper log files, zookeeper.out and /opt/zookeeper/log/zkMonitor.log, are not rotated, when new logs are added. This can cause disk ...
Read more >Troubleshooting: Disks Filling Up - Humio Documentation
Disks used by Humio fill up with data. Humio runs out of disk space. Disk space usage increases and space is not recovered....
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
LGTM
This is fixed by apache/bookkeeper#2097