Bookkeeper containers are keep on restarting after deploying pravega with 3 zookeepers in docker swarm setup

Deploy pravega:0.5.0-2260.b477eec release in docker swarm setup with 3 zookeeper containers

1 hdfs
3 zookeepers
1 segmentstore
1 controller
4 bookkeepers

First deployed zookeeper with 3 replicas. After all three zookeeper came up deployed pravega, but while installing bookkeeper all 4 bookkeepers keep on restarting and never came up.

Same issue observed while verifying in single node(1 master) setup and multiple node (1 master & 2 worker)

Getting below exception while restarting the containers (logs from Exited bookie containers)

2019-06-04 12:10:18,670 - INFO  - [main-EventThread:ZooKeeperWatcherBase@130] - ZooKeeper client is connected now.
2019-06-04 12:10:18,686 - ERROR - [main:ZKRegistrationManager@374] - BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat
2019-06-04 12:10:18,687 - INFO  - [main:BookieNettyServer@396] - Shutting down BookieNettyServer
2019-06-04 12:10:18,728 - ERROR - [main:Main@221] - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
        at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:387)
        at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:413)
        at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:257)
        at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:641)
        at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:131)
        at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:100)
        at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43)
        at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:299)
        at org.apache.bookkeeper.server.Main.doMain(Main.java:219)
        at org.apache.bookkeeper.server.Main.main(Main.java:201)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata
        at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:377)
        ... 9 more

NOTE :- while scale down zookeeper from 3 to 2, all 4 bookkeepers came up

[Ranjan-Padhi]

Thanks @shrids ,

I have tried with ZOO_SERVERS and ZOO_MY_ID configuration in zookeeper yml file and it’s working properly

After deploying zookeeper with new below zookeeper yml file, pravega bookies deployed successfully without restarting and IO also running successfully

Zookeeper.yml file

version: '3.1'

services:
  zookeeper1:
    image: zookeeper:3.5.4-beta
    hostname: zookeeper1
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zookeeper2:2888:3888;2181 server.3=zookeeper3:2888:3888;2181

  zookeeper2:
    image: zookeeper:3.5.4-beta
    hostname: zookeeper2
    ports:
      - 2182:2181
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zookeeper1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zookeeper3:2888:3888;2181

  zookeeper3:
    image: zookeeper:3.5.4-beta
    hostname: zookeeper3
    ports:
      - 2183:2181
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zookeeper1:2888:3888;2181 server.2=zookeeper2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181

[shrids]

Please use https://hub.docker.com/_/zookeeper and https://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperStarted.html#sc_RunningReplicatedZooKeeper as a reference to deploy a zk cluster in replicated mode.

(Just changing to replicas: 3 will cause all the zk instances to be deployed as separate individual instances and the request to zookeeper:2181 could be serviced by any of the individual zk instance. )