Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Failed to get broker metrics for BrokerIdentity

See original GitHub issue

I failed to see the data size from brokers’ metrics. The JMX port should worked properly as I can see it from this metrics and the target port is listened as well. I have checked some similar issues but still cannot find a solution in my case. Could you help me working this out?

Below is the error message:

[error] k.m.a.c.BrokerViewCacheActor - Failed to get broker metrics for BrokerIdentity(2,10.13.0.93,9092,9997)
java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[na:1.8.0_71]
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.8.0_71]
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_71]
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130) ~[na:1.8.0_71]
    at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) ~[na:1.8.0_71]
    at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) ~[na:1.8.0_71]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) ~[na:1.8.0_71]
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_71]
    at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:50) ~[kafka-manager.kafka-manager-1.3.0.7-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerMetrics$1$$anonfun$apply$27$$anonfun$apply$3.apply$mcV$sp(BrokerViewCacheActor.scala:358) ~[kafka-manager.kafka-manager-1.3.0.7-sans-externalized.jar:na]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_71]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_71]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_71]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_71]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_71]
    at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_71]
    at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_71]
    at java.net.Socket.<init>(Socket.java:434) ~[na:1.8.0_71]
    at java.net.Socket.<init>(Socket.java:211) ~[na:1.8.0_71]
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_71]

Issue Analytics

State:
Created 7 years ago
Comments:8

Top GitHub Comments

7reactions

captainbuptcommented, Mar 25, 2016

I fixed this issues by adding --override -Djava.rmi.server.hostname=${ip} when start kafka brokers. It turns out that the JMX will publish the hostname as 127.0.0.1 by default and we have to specify the hostname by ourselves. There will be no problem if you do have hostnames for your servers.

3reactions

captainbuptcommented, Mar 6, 2017

My final commands is KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${ip}" JMX_PORT=9997 /data0/kafka/bin/kafka-server-start.sh /data0/kafka/config/server.properties