exposing kafka from docker

I’m using the Docker setup provided in this repo, and was wondering if anyone has tried to access the scraped data in the dockerized Kafka from outside of the container.

The docker-compose.yml file does expose the right ports, yet I couldn’t appear to subscribe to topics from outside the container, though from inside this worked well (using the same command).

This seemed like a Kafka + Docker problem, so I googled around, and most answers involved setting the KAFKA_ADVERTISED_HOST_NAME environment variable, currently absent from docker-compose.yml.

I tried setting that to different values (127.0.0.1 or localhost; kafka or scrapycluster_kafka_1 with an entry in my hosts file to route them to 127.0.0.1), but any of these would mess up the communication between the different containers for this repo – to the point where I also wasn’t able to verify whether the change had its intended effect.

At this point I’d wonder, has anyone else managed to interface with data scraped using the provided Docker Compose file from outside the container?

Steps to reproduce:

docker compose up
/opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --list
# ^ ok, works from both inside and from outside container
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic demo.incoming
curl 127.0.0.1:5343/feed -H "Content-Type: application/json" -d '{"url":"http
://www.google.com", "appid":"curl", "crawlid":"abc", "allowed_domains":["www.google.com"]}'
# ^ listen to topic and ensure it has messages -- successfully shows message from kafka container, not from host
^C
nano docker-compose.yml
# add environment variable for Kafka: KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
docker compose up
# bunch of errors:
# rest_1           | 2017-10-27 10:28:18,226 [rest-service] ERROR: Couldn't initialize kafka consumer for topic
# kafka_1          | waiting for kafka to be ready
...
# kafka_1          | [2017-10-27 10:28:25,891] WARN Unexpected error from /172.18.0.1; closing connection (org.apache.kafka.common.network.Selector)
# kafka_1          | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
...
# rest_1           | 2017-10-27 10:28:58,742 [rest-service] ERROR: Failed to send message to Kafka

[gsunjka]

@tycho01

I had the same issue. After adding my host ip address to KAFKA_ADVERTISED_HOST_NAME everything worked fine.

[User1m]

thanks @gsunjka this helped solve my issue. in case you need to find the host ip address from dockers point run /sbin/ip route|awk '/default/ { print $3 }' inside a container