Support swarm mode and swarm service scaling

This is a request to support Docker swarm mode (in 1.12+).

The current image can already be used with swarm mode i.e. you would create the service similar to the following. This creates services on specific nodes so that a given kafka service is started, it always starts on the same node with the same data volume.

docker network create --driver overlay --attachable zookeeper
docker network create --driver overlay --attachable kafka

# Tie kafka instances to specific nodes
docker node update --label-add kafka=1 node1
docker node update --label-add kafka=2 node2
docker node update --label-add kafka=3 node3

// create zookeeper-1, zookeeper-2, zookeeper-3 services here

docker service create \
  --name kafka-1 \
  --network kafka \
  --network zookeeper \
  --restart-condition on-failure \
  --restart-max-attempts 3 \
  --log-driver=json-file \
  --constraint "node.role != manager" \
  --constraint "node.labels.kafka == 1" \
  --mount type=volume,src=kafka_vol,target=/var/lib/kafka/data \
  --env KAFKA_BROKER_ID=1 \
  --env KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 \
  --env KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-1:9092 \
  confluentinc/cp-kafka:3.1.2

sleep 60

docker service create \
  --name kafka-2 \
  --network kafka \
  --network zookeeper \
  --restart-condition on-failure \
  --restart-max-attempts 3 \
  --log-driver=json-file \
  --constraint "node.role != manager" \
  --constraint "node.labels.kafka == 2" \
  --mount type=volume,src=kafka_vol,target=/var/lib/kafka/data \
  --env KAFKA_BROKER_ID=2 \
  --env KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 \
  --env KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-2:9092 \
  confluentinc/cp-kafka:3.1.2

sleep 60

docker service create \
  --name kafka-3 \
  --network kafka \
  --network zookeeper \
  --restart-condition on-failure \
  --restart-max-attempts 3 \
  --log-driver=json-file \
  --constraint "node.role != manager" \
  --constraint "node.labels.kafka == 3" \
  --mount type=volume,src=kafka_vol,target=/var/lib/kafka/data \
  --env KAFKA_BROKER_ID=3 \
  --env KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 \
  --env KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-3:9092 \
  confluentinc/cp-kafka:3.1.2

This works quite well, but it does not support the docker swarm mode scale option, or similar capabilities like rolling updates, because each kafka instance is its own service e.g.:

docker service scale kafka=n

This is a request to support docker swarm mode and docker swarm scaling, which makes manageability a lot simpler. Some of the relevant problems that need to be solved are:

• naming kafka services uniquely for registry in zookeeper, but still allowing individual kafka services to be referenceable (the swarm-created DNS name tasks.<servicename> will be useful for this)
• giving each service a unique broker id
• supporting/testing shared volume mechanisms like flocker

[vldocker]

@chopanpma can you share how to deploy the stack in swarm? thanks

[aayars]

Hi @rocketraman and @FryDerm,

Right now, Confluent does not plan to officially support swarm in our Docker images. That said, we offer a hearty “go for it!”