Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Several issues with the kubernetes generic example

See original GitHub issue

Expected behavior

It should be able to be adjusted to run more then one bookie.

Actual behavior

When you adjust the kubernetes example to run more then one bookie without multiple kubernetes nodes, the second bookie will crash with There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current] now it’s just on bookie-0 instead of bookie-1`

Steps to reproduce

Change the DaemonSet to be a Statefulset, and change replicas to 2

System configuration

Pulsar version: 2.2.0

More info

Debugged together with @ivankelly. Several suggestions:

change from DaemonSet to Statefulset as in the GKE example
remove the environment to set the advertisedAddress from the hostIP, let it just use it’s POD ip (like the GKE example)
Remove the initContainer as there is a job to initialize the cluster metadata
Change the container to wait for zookeeper to be up (apparently there is a script for this), as this removes the need to spin up zookeeper first as the documentation suggests.
have PVC’s for all zookeeper and bookie instances (not really part of the issue)

Issue Analytics

State:
Created 5 years ago
Comments:15 (10 by maintainers)

Top GitHub Comments

4reactions

sijiecommented, Jun 9, 2020

Since we have deprecated the generic k8s yaml files and moved to helm chart, please try the helm chart. Close this issue for now. If you have any issues related to helm chart, please let us know.

2reactions

sijiecommented, Jul 31, 2020

Okay. What storage provisioner are you using? The error basically says a bookie attempts to start with empty directories. It means your disks are reformatted. Bookie prevents startup with this corrupted state. So it would be good to understand why the disks are reformatted.

To repair it, you can run decommissionbookie to decommission the bookie first. The decommission command will re-replicate the data and delete the cookie in the zookeeper. So the bookie can restart freshly.

But anyway the most important thing is to figure out why the bookie starts with empty directory.

Top Results From Across the Web

[GitHub] pvanderlinden opened a new issue #3121

pvanderlinden opened a new issue #3121: Several issues with the kubernetes generic example URL: https://github.com/apache/pulsar/issues/3121 ...

Troubleshooting Clusters | Kubernetes

VM(s) shutdown · Network partition within cluster, or between cluster and users · Crashes in Kubernetes software · Data loss or unavailability of ......

Generically working with Kubernetes objects in Go - ITNEXT

Some example use-cases are: Working with k8s API objects from a plugin that does not have associated Golang structs. Performing generic CRUD ...

Automatic Remediation of Kubernetes Nodes

Limiting automatic remediation to generic worker nodes is important as there are other node types in our clusters where more care is required....

Troubleshooting 'terminated with exit code 1' error - ContainIQ

Unexpected Kubernetes pod terminations can be frustrating when you're left with a vague error message. Unclear root causes often delay remediation, ...