Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Cannot connect to a broker on another machine (LAN, no Docker/Cloud)
See original GitHub issueFew month i was developing the Kafka application with broker and client on the same machine. Then I gave it to QA and he said that it is not consuming or producing. I tested it out my application with another machine and it didn’t work too. Then I tried to test it without my application and without docker just using Confluent (1.0.0) inside console application and Apache Kafka broker (kafka_2.12-2.2.0) on Windows and it failed…
When I try to produce message on client I receive “No connection could be made because the target machine actively refused it”
And the log file on broker
2020-01-20 16:48:50,101] WARN [SocketServer brokerId=0] Unexpected error from /172.17.0.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:104)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at kafka.network.Processor.poll(SocketServer.scala:830)
at kafka.network.Processor.run(SocketServer.scala:730)
at java.lang.Thread.run(Thread.java:748)
However I can fetch metadata successfully
adminClient.GetMetadata(TimeSpan.FromSeconds(10));
producerConfig
var config = new ProducerConfig
{
BootstrapServers = "192.168.1.75:9092",
MessageTimeoutMs = 2000,
Debug = "all"
};
server.properties
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# SASL with SSL encryption
listeners=PLAINTEXT://:9092,SASL_SSL://:9093
advertised.listeners=PLAINTEXT://127.0.0.1:9092,SASL_SSL://127.0.0.1:9093
ssl.client.auth=required
ssl.key.password=fastdata
ssl.keystore.location=certs/kafka.jks
ssl.keystore.password=fastdata
ssl.truststore.location=certs/truststore.jks
ssl.truststore.password=fastdata
ssl.protocol=TLS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS
# List of enabled mechanisms, can be more than one
sasl.enabled.mechanisms=PLAIN
# Specify one of of the SASL mechanisms
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.jaas.config=config/kafka_server_jaas.conf
#sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
IP of the broker machine is 192.168.1.75 IP of the client machine is 192.168.1.26
I tried to change advertised.listeners to advertised.listeners=PLAINTEXT://192.168.1.75:9092,SASL_SSL://127.0.0.1:9093
and now I receive
0-01-21 19:35:57,794] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-48 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2020-01-21 19:35:59,235] WARN [SocketServer brokerId=0] Unexpected error from /192.168.1.26; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295617 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:104)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:640)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:561)
at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
at kafka.network.Processor.poll(SocketServer.scala:830)
at kafka.network.Processor.run(SocketServer.scala:730)
at java.lang.Thread.run(Thread.java:748)
[2020-01-21 19:36:04,590] WARN [SocketServer brokerId=0] Unexpected error from /192.168.1.26; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295617 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:104)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:640)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:561)
at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
at kafka.network.Processor.poll(SocketServer.scala:830)
at kafka.network.Processor.run(SocketServer.scala:730)
at java.lang.Thread.run(Thread.java:748)
and producer no longer throw with “No connection could be made because the target machine actively refused it”, but is stuck in loop
%7|1579627238.901|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1579627238.901|WAKEUPFD|rdkafka#producer-1| [thrd:app]: 192.168.1.75:9092/bootstrap: Enabled low-latency ops queue wake-ups
%7|1579627238.901|BROKER|rdkafka#producer-1| [thrd:app]: 192.168.1.75:9092/bootstrap: Added new broker with NodeId -1
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:app]: 192.168.1.75:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1579627238.902|BRKMAIN|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Enter main broker thread
%7|1579627238.902|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.0.0 (0x10000ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0xffff)
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Received CONNECT op
%7|1579627238.902|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1579627238.902|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1579627238.902|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1579627238.902|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Connecting to ipv4#192.168.1.75:9092 (plaintext) with socket 712
%7|1579627238.905|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Connected to ipv4#192.168.1.75:9092
%7|1579627238.906|CONNECTED|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Connected (#1)
%7|1579627238.906|FEATURE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1579627238.906|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1579627238.906|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627238.907|SEND|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1)
%7|1579627238.916|TOPIC|rdkafka#producer-1| [thrd:app]: New local topic: test-topic
%7|1579627238.916|TOPPARNEW|rdkafka#producer-1| [thrd:app]: NEW test-topic [-1] 00E37FC0 (at rd_kafka_topic_new0:393)
%7|1579627239.900|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic metadata information unknown
%7|1579627239.901|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic partition count is zero: should refresh metadata
%7|1579627239.902|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1579627239.903|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
%7|1579627239.903|METADATA|rdkafka#producer-1| [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
%7|1579627239.904|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
%7|1579627240.905|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic metadata information unknown
%7|1579627240.905|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic partition count is zero: should refresh metadata
%7|1579627240.907|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1579627240.908|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
%7|1579627240.908|METADATA|rdkafka#producer-1| [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
%7|1579627240.908|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
%7|1579627241.909|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic metadata information unknown
%7|1579627241.909|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic partition count is zero: should refresh metadata
%7|1579627241.911|TIMEOUT|rdkafka#producer-1| [thrd:main]: test-topic: 1 message(s) timed out
%7|1579627241.911|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1579627241.912|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
%7|1579627241.912|METADATA|rdkafka#producer-1| [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
%7|1579627241.912|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
Delivery failed: Local: Message timed out
%7|1579627242.015|DESTROY|rdkafka#producer-1| [thrd:app]: Terminating instance (destroy flags none (0x0))
%7|1579627242.015|TERMINATE|rdkafka#producer-1| [thrd:app]: Interrupting timers
%7|1579627242.017|TERMINATE|rdkafka#producer-1| [thrd:app]: Sending TERMINATE to internal main thread
%7|1579627242.018|TERMINATE|rdkafka#producer-1| [thrd:app]: Joining internal main thread
%7|1579627242.018|TERMINATE|rdkafka#producer-1| [thrd:main]: Internal main thread terminating
%7|1579627242.019|DESTROY|rdkafka#producer-1| [thrd:main]: Destroy internal
%7|1579627242.019|BROADCAST|rdkafka#producer-1| [thrd:main]: Broadcasting state change
%7|1579627242.020|DESTROY|rdkafka#producer-1| [thrd:main]: Removing all topics
%7|1579627242.020|TOPPARREMOVE|rdkafka#producer-1| [thrd:main]: Removing toppar test-topic [-1] 00E37FC0
%7|1579627242.021|DESTROY|rdkafka#producer-1| [thrd:main]: test-topic [-1]: 00E37FC0 DESTROY_FINAL
%7|1579627242.021|DESTROY|rdkafka#producer-1| [thrd:main]: Sending TERMINATE to 192.168.1.75:9092/bootstrap
%7|1579627242.021|TERMINATE|rdkafka#producer-1| [thrd:main]: Purging reply queue
%7|1579627242.022|TERMINATE|rdkafka#producer-1| [thrd:main]: Decommissioning internal broker
%7|1579627242.022|TERMINATE|rdkafka#producer-1| [thrd:main]: Join 2 broker thread(s)
%7|1579627242.021|TERM|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Received TERMINATE op in state APIVERSION_QUERY: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 1 waitresps, 0 retrybufs
%7|1579627242.022|TERM|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
%7|1579627242.022|BROKERFAIL|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.023|BROKERFAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.023|FAIL|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Client is terminating (after 3117ms in state APIVERSION_QUERY)
%7|1579627242.023|FAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Client is terminating (after 3123ms in state INIT)
%7|1579627242.024|FEATURE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updated enabled protocol features -ApiVersion to
%7|1579627242.025|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state APIVERSION_QUERY -> DOWN
%7|1579627242.024|STATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Broker changed state INIT -> DOWN
%7|1579627242.028|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627242.028|BROADCAST|rdkafka#producer-1| [thrd::0/internal]: Broadcasting state change
%7|1579627242.028|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 1 buffers
%7|1579627242.029|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.029|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 0 buffers
%7|1579627242.030|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.030|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updating 0 buffers on connection reset
%7|1579627242.030|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Updating 0 buffers on connection reset
%7|1579627242.031|TERMINATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Handle is terminating in state DOWN: 1 refcnts (00DDE048), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1579627242.030|TERMINATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Handle is terminating in state DOWN: 1 refcnts (00DE1D28), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1579627242.031|BROKERFAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.031|BROKERFAIL|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.032|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.032|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 0 buffers
%7|1579627242.032|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.033|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Updating 0 buffers on connection reset
%7|1579627242.033|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 0 buffers
%7|1579627242.035|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updating 0 buffers on connection reset
%7|1579627242.035|TERMINATE|rdkafka#producer-1| [thrd:main]: Internal main thread termination done
%7|1579627242.036|TERMINATE|rdkafka#producer-1| [thrd:app]: Destroying op queues
%7|1579627242.038|TERMINATE|rdkafka#producer-1| [thrd:app]: Termination done: freeing resources
What am I doing wrong? Why does everything work fine when client and broker are on localhost? Why is metadata fetched without errors, but producing and consuming doesn’t connect? Please help
[ 1.0.0 ] Confluent.Kafka nuget version.
Issue Analytics
- State:
- Created 4 years ago
- Comments:5 (1 by maintainers)
Top Results From Across the Web
Docker container has access to internet but not the local ...
The container has access to the internet (it can access 3rd party nodes external to the network), but it is unable to connect...
Read more >Get started with macvlan network driver - Docker
In Macvlan mode, containers on separate networks cannot reach one another ... For example, Use the same subnet and gateway of the Docker...
Read more >Self-Host All the Things?
I have a rack with 10gbe, ups, kubernetes a zfs storage server, multiple vlans, 4 unifi APs & locally hosted controller and all...
Read more >Error Docker Cloud Agent - Ask a question
Hello everyone! I ever use a Docker Cloud Agent in my Jenkins server, but recently I do some updates and the communication with...
Read more >ROS node inside docker not being listened to by external ...
Coding example for the question ROS node inside docker not being listened to by external mchines on same LAN-docker.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
this may be relevant: https://rmoff.net/2018/08/02/kafka-listeners-explained/
Seems to be related https://github.com/wurstmeister/kafka-docker/issues/100#issuecomment-297815839