Kafka Manager API timing out

I have deployed kafka manager in several environments, while it works in all non production environments it keeps timing out when we try to access any topic level information in the production environment.

2015-10-21 17:16:46,276 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2 
error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [90000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [90000 ms]
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[org.scala-lang.scala-library-2.11.7.jar:na]
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[org.scala-lang.scala-library-2.11.7.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[org.scala-lang.scala-library-2.11.7.jar:na]
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]

• I can manually connect to kafka and zookeeper fine from the box where kafka manager is installed. I have also seen kafka manager to work a couple of times without timeouts.
• I’m trying to connect to a 10 node kafka cluster + 5 node zookeeper cluster.
• Kafka manager also gets into a stuck state if I try to update the cluster information.
• Also tried with a timeout of 2 mins as against the default 5 secs, but no joy.

I can’t seem to understand why there would be timeouts especially since the kafka manager and kafka nodes are all in the same internal network. Any pointers in finding the root cause?

[radutanasa]

I’m experiencing the exact same problem and unfortunately enabling the active offsetcache doesn’t fix it. Is there another workaround?

[josephfrancis]