Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Failed to build bookie server in kubernetes environment

See original GitHub issue

Describe the bug we have deployed pulsar using helm chart in openshfit k8s environment. when we scale up the deleted volume bookie we encounter with below issue.

2.284: Total time for which application threads were stopped: 0.0001663 seconds, Stopping threads took: 0.0001040 seconds
10:28:55.441 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]
10:28:55.441 [main] INFO  org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
2.404: Total time for which application threads were stopped: 0.0002793 seconds, Stopping threads took: 0.0001671 seconds
2.405: Total time for which application threads were stopped: 0.0002462 seconds, Stopping threads took: 0.0001874 seconds
2.406: Total time for which application threads were stopped: 0.0002902 seconds, Stopping threads took: 0.0001068 seconds
2.413: Total time for which application threads were stopped: 0.0005099 seconds, Stopping threads took: 0.0003770 seconds
2.413: Total time for which application threads were stopped: 0.0001357 seconds, Stopping threads took: 0.0000875 seconds
2.415: Total time for which application threads were stopped: 0.0004179 seconds, Stopping threads took: 0.0001708 seconds
2.417: Total time for which application threads were stopped: 0.0001661 seconds, Stopping threads took: 0.0000410 seconds
2.418: Total time for which application threads were stopped: 0.0003534 seconds, Stopping threads took: 0.0002680 seconds
2.419: Total time for which application threads were stopped: 0.0003882 seconds, Stopping threads took: 0.0003552 seconds
2.419: Total time for which application threads were stopped: 0.0000419 seconds, Stopping threads took: 0.0000138 seconds
2.420: Total time for which application threads were stopped: 0.0000399 seconds, Stopping threads took: 0.0000124 seconds
10:28:55.461 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: 
	at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:468) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:688) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:41) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
	at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
2.425: Total time for which application threads were stopped: 0.0000542 seconds, Stopping threads took: 0.0000156 seconds

To Reproduce Steps to reproduce the behavior :

scale down the bookie pod.
delete persistent volume claim and persistent volume of deleted bookie pod.
scale up the bookie pod

Expected behavior scale up the bookie pod should not cause the problem.

Issue Analytics

State:
Created 3 years ago
Comments:7 (3 by maintainers)

Top GitHub Comments

1reaction

walterEricommented, Jul 27, 2020

In my scenario i have 3 bookie and each pod attached to separate pv and pvc. Then i scaled down to 2 pod and deleted pv and pvc for the 3rd pod. so we don’t have OLD volume of 3rd pod.

Then again i have scaled up to 3 bookie. so it’s creating a new pod (3rd) and attaching new pv and pvc to the 3rd pod. even though i am getting the same error.

0reactions

truong-huacommented, May 3, 2022

Let’s say that in an unexpected accident that we lost all of the bookie data. Is there any option to force delete relative ledgers and force decommission the bookie?

Top Results From Across the Web

subject:"\[GitHub\] \[pulsar\] walterEri commented ... - The Mail Archive

[GitHub] [pulsar] walterEri commented on issue #7653: Failed to build bookie server in kubernetes environment · 2020-07-31 Thread GitBox.

Broker pod failed unable to connect to Zookeeper - Micro Focus

Run the following command to stop the pods: · Ensure all the pods of opsb suite namespace are terminated by running - #kubectl...

Troubleshooting Apache Pulsar Issues | Fusion 5.6

The org.apache.bookkeeper.bookie.BookieException$InvalidCookieException error is typically generated when corrupted or lost data exists on the persistent ...

I cannot get a Pulsar cluster to restart properly - Stack Overflow

kubectl logs pod-name --all-containers=true ... You can check whether your liveness probes are failing which is resulting in crashes.

BookKeeper administration

A typical BookKeeper installation consists of an ensemble of bookies and a ... create a log4j.properties file and point the BOOKIE_LOG_CONF environment ......