Kafka unable to connect to zookeeper in a secluded environment

Hi,

I am running the Oeprator in a secured GKE and i get the below error, where kafka is not able to connect with zookeeper.

for the below error it seems the kafka is trying to zookeeper connect on localhost??? INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [main-SendThread(localhost:2181

could you advice how to alter the hooks/ if anyother issues what can be done to work this?

ibs/kafka-agent.jar:/opt/kafka/bin/../libs/kafka-clients-2.3.1.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.3.1.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.4.0.jar
/opt/kafka/bin/../libs/kafka-oauth-common-0.4.0.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.4.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.4.0.jar:/op
/kafka/bin/../libs/kafka-streams-2.3.1.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.3.1.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.12-2.3.1.jar:/opt/kafka/bin/
./libs/kafka-streams-test-utils-2.3.1.jar:/opt/kafka/bin/../libs/kafka-tools-2.3.1.jar:/opt/kafka/bin/../libs/kafka_2.12-2.3.1-sources.jar:/opt/kafka/bin/../libs/kafka_2.12
2.3.1.jar:/opt/kafka/bin/../libs/keycloak-common-7.0.0.jar:/opt/kafka/bin/../libs/keycloak-core-7.0.0.jar:/opt/kafka/bin/../libs/libthrift-0.12.0.jar:/opt/kafka/bin/../libs
log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../l
bs/mirror-maker-agent.jar:/opt/kafka/bin/../libs/okhttp-3.9.0.jar:/opt/kafka/bin/../libs/okio-1.13.0.jar:/opt/kafka/bin/../libs/opentracing-api-0.33.0.jar:/opt/kafka/bin/..
libs/opentracing-kafka-client-0.1.4.jar:/opt/kafka/bin/../libs/opentracing-noop-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-tracerresolver-0.1.8.jar:/opt/kafka/bin/../lib
/opentracing-util-0.33.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.0.jar:/
pt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/bin/../libs/scala-library-2.12.10.jar:/opt/kafka/bin/../libs/scala-libra
y-2.12.8.jar:/opt/kafka/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka/bin/
./libs/slf4j-log4j12-1.7.26.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka/bin/../libs/tracing-agent.ja
:/opt/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/bin/../libs/zkclient-0.11.jar:/opt/kafka/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka/bin/../libs/zstd-jni-
.4.0-1.jar:/opt/kafka/libs/kafka-agent.jar (org.apache.zookeeper.ZooKeeper) [main]                                                                                          
2020-04-08 15:01:26,313 INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [main]      
2020-04-08 15:01:26,313 INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [main]                                                                 
2020-04-08 15:01:26,313 INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [main]                                                                  
2020-04-08 15:01:26,313 INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [main]                                                                       
2020-04-08 15:01:26,313 INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [main]                                                                       
2020-04-08 15:01:26,313 INFO Client environment:os.version=4.19.102+ (org.apache.zookeeper.ZooKeeper) [main]                                                                
2020-04-08 15:01:26,313 INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper) [main]                                                                     
2020-04-08 15:01:26,313 INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper) [main]                                                               
2020-04-08 15:01:26,313 INFO Client environment:user.dir=/opt/kafka (org.apache.zookeeper.ZooKeeper) [main]                                                                 
2020-04-08 15:01:26,315 INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@
722c3c3 (org.apache.zookeeper.ZooKeeper) [main]                                                                                                                             
2020-04-08 15:01:26,329 INFO Starting poller (io.strimzi.kafka.agent.KafkaAgent) [main]                                                                                     
2020-04-08 15:01:26,331 INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper
ClientCnxn) [main-SendThread(localhost:2181)]                                                                                                                               
2020-04-08 15:01:26,332 INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [main]                                               
2020-04-08 15:01:26,336 INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [main-SendThread(localhost:2181
]                                                                                                                                                                           
2020-04-08 15:01:32,335 INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient) [main]                                                               
2020-04-08 15:01:32,338 WARN Client session timed out, have not heard from server in 6001ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn) [main-SendThread(localhost:2
81)]                                                                                                                                                                        
2020-04-08 15:01:32,444 INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [main]                                                                                    
2020-04-08 15:01:32,444 INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn) [main-EventThread]                                                    
2020-04-08 15:01:32,446 INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient) [main]                                                                
2020-04-08 15:01:32,450 ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) [main]                                                 
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING                                                                
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)                                                                         
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)                                                                                           
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)                                                                                                               
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)                                                                                    
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:112)                                                                                                
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1826)                                                                                                          
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:364)                                                                                                 
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:387)                                                                                                     
        at kafka.server.KafkaServer.startup(KafkaServer.scala:207)                                                                                                          
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)                                                                                         
        at kafka.Kafka$.main(Kafka.scala:84)                                                                                                                                
        at kafka.Kafka.main(Kafka.scala)                                                                                                                                    
2020-04-08 15:01:32,452 INFO shutting down (kafka.server.KafkaServer) [main]                                                                                                
2020-04-08 15:01:32,459 INFO shut down completed (kafka.server.KafkaServer) [main]                                                                                          
2020-04-08 15:01:32,460 ERROR Exiting Kafka. (kafka.server.KafkaServerStartable) [main]                                                                                     
2020-04-08 15:01:32,462 INFO shutting down (kafka.server.KafkaServer) [kafka-shutdown-hook]                                                                                 

Help Please!!!

[scholzj]

Oh, sorry. You should run omething like this: report.sh --namespace=<namespace> --cluster=<cluster> where <cluster> is the name of your Kafka CR and <namespace> is the namespace where the cluster is installed.

[malikbenkirane]

The issue was internal to GCP. it fixed without our fix.

any chance you can give us more insights on this internal issue to GCP

On my side the entity operator always restart complaining

2021-12-22 07:18:30,96606 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2021-12-22 07:18:30,96757 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2021-12-22 07:18:30,96891 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) [org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]

The logs above are coming from

On the other hand, that would be really valuable to have some doc specific to this context.

Running gke-1.22 autopilot private cluster