question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes

See original GitHub issue

Describe the bug A clear and concise description of what the bug is. pulsar 2.7.1

 k get pod -n pulsar 
NAME                                     READY   STATUS             RESTARTS   AGE
pulsar-bookie-0                          0/1     CrashLoopBackOff   73         5h58m
pulsar-bookie-1                          0/1     CrashLoopBackOff   73         5h58m
pulsar-bookie-2                          0/1     CrashLoopBackOff   73         5h58m
pulsar-bookie-3                          0/1     CrashLoopBackOff   73         5h58m
pulsar-broker-0                          0/1     CrashLoopBackOff   71         5h58m
pulsar-broker-1                          0/1     CrashLoopBackOff   71         5h58m
pulsar-broker-2                          0/1     CrashLoopBackOff   71         5h57m
pulsar-proxy-0                           1/1     Running            0          5h57m
pulsar-proxy-1                           1/1     Running            0          5h57m
pulsar-proxy-2                           1/1     Running            0          5h57m
pulsar-pulsar-manager-6bf99b4d88-jk22g   1/1     Running            0          27d
pulsar-recovery-0                        1/1     Running            1          27d
pulsar-toolset-0                         1/1     Running            1          27d
pulsar-zookeeper-0                       1/1     Running            1          27d
pulsar-zookeeper-1                       1/1     Running            1          27d
pulsar-zookeeper-2                       1/1     Running            1          27d

bookie

k logs -n pulsar  pulsar-bookie-0 --tail 20 -f 
22:20:20.891 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
22:20:20.895 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
22:20:20.904 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
22:20:20.918 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/172.16.35.208:2181. Will not attempt to authenticate using SASL (unknown error)
22:20:20.924 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.16.67.24:45748, server: pulsar-zookeeper/172.16.35.208:2181
22:20:20.931 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/172.16.35.208:2181, sessionid = 0x2000001f885027b, negotiated timeout = 30000
22:20:20.934 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
22:20:21.104 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current]
22:20:21.104 [main] INFO  org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
22:20:21.113 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: 
	at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:476) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:712) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
	at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]

broker

22:28:45.687 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 Internal Server Errorjava.io.ByteArrayInputStream@7c3c22f9

so is there anyone who can help me out?

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: centos 7.9 Additional context Add any other context about the problem here.

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
zymapcommented, May 26, 2021

Yes. It should fix the issue.

0reactions
codelipenghuicommented, Mar 4, 2022

The issue had no activity for 30 days, mark with Stale label.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Fail to install GoodData in a Self-Hosted Kubernetes cluster
Hello,I'm working in a French company, and an R&D developer did a demo of GoodData with your Docker version.The demo was positive and...
Read more >
Failed to create consumer: Subscription is fenced
This issue is because the COSO Message Bus topic consumer got disconnected and reconnected. The reconnection may happen before the consumer ...
Read more >
grcp [statuscode.unavailable] socket closed - You.com
... priority and fairness, request: GET:https://kubernetes.default.svc.cluster.l ... <none> <none> benchmark-backup-7bhxx-1-pulsar-bookie-0 1/1 Running 0 9h ...
Read more >
There is a probability that the installation will not be successful ...
[2022-05-31T01:59:17.016Z] + kubectl get pods [2022-05-31T01:59:17.016Z] + grep ... 0 9m19s [2022-05-31T01:59:17.593Z] cluster-reinstall-649-pulsar-bookie-0 ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found