Connection to node -1 could not be established. Broker may not be available.

This issue follows discussion: https://discuss.lightbend.com/t/error-running-the-reference-hello-lagom-application-for-scala/265

Sometimes, after a crash of the application, a restart using sbt runAll produces the following error:

[warn] org.apache.kafka.clients.NetworkClient [] - Connection to node -1 could not be established. Broker may not be available.

which appears continually on logging console.

The first time I saw this error was with online-auction-scala example; my new code based on it produces this error very often (at least once a day). I don’t have yet a trivial reproducer (the most similar procedure seems to be to run this example and to stop it with something like Ctrl-C to emulate an application crash and then restart the application).

At the runAllstartup, Kafka crashes:

[info] Starting Kafka
[info] Starting Cassandra
..........
[info] Cassandra server running at 127.0.0.1:4000
Kafka Server closed unexpectedly.

If I inspect logs in the directory you suggest, I can see in server.log (from target/lagom-dynamic-projects/lagom-internal-meta-project-kafka/target/log4j_outputdirectory``` – full log as attached file)

[2018-03-26 14:07:07,953] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2018-03-26 14:07:07,957] INFO Got user-level KeeperException when processing sessionid:0x16262346bea0000 type:create cxid:0x24 zxid:0xbe txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NodeExists for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
[2018-03-26 14:07:07,957] INFO Got user-level KeeperException when processing sessionid:0x16262346bea0000 type:create cxid:0x25 zxid:0xbf txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = NodeExists for /brokers/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2018-03-26 14:07:07,957] INFO Got user-level KeeperException when processing sessionid:0x16262346bea0000 type:create cxid:0x26 zxid:0xc0 txntype:-1 reqpath:n/a Error Path:/brokers/ids/0 Error:KeeperErrorCode = NodeExists for /brokers/ids/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2018-03-26 14:07:07,960] INFO Result of znode creation is: NODEEXISTS (kafka.utils.ZKCheckedEphemeral)
[2018-03-26 14:07:07,961] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
[server.log](https://github.com/lagom/lagom/files/1848163/server.log)

Deleting target/lagom-dynamic-projects/lagom-internal-meta-project-kafka/target/zookeeper_data directory seems to help to clean embedded Kafka data and to restart without this error.

If this error is related to a corruption of Zookeeper data, is there a way to to prevent this corruption or to automacally repair the embedded Kafka while using runAll sbt command ?

NB: My configuration is:

• Scala environment
• Lagom 1.4.1
• Java 1.8.0.144
• macOS High Sierra 10.13.3

[TimMoore]

@deepakmehra10 this usually happens because Kafka cannot start up, which could happen for various reasons. Please review the Kafka logs for the reason.

[dcguim]

This issue normally appears when listeners or advertised.listeners on config/server.properties are not appropriately configured or used. More, info