Automating the cluster setup

I am incorporating kafka-manager in a kafka/zookeeper openshift/kubernetes config, and wish to automate the cluster setup.

I am using a docker image similar to sheepkiller/kafka-manager-docker, whereby I am passing an env var for the zookeeper connection. Anyhow I am seeing kafka-manager connecting to zookeeper fine:

[info] o.a.z.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@5c9fdc3a
[info] k.m.a.KafkaManagerActor - zk=zookeeper:2181
[info] k.m.a.KafkaManagerActor - baseZkPath=/kafka-manager
[info] o.a.z.ClientCnxn - Opening socket connection to server zookeeper.kafka.svc.cluster.local/172.30.151.6:2181. Will not attempt to authenticate using SASL (unknown error)
[info] o.a.z.ClientCnxn - Socket connection established to zookeeper.kafka.svc.cluster.local/172.30.151.6:2181, initiating session
[info] o.a.z.ClientCnxn - Session establishment complete on server zookeeper.kafka.svc.cluster.local/172.30.151.6:2181, sessionid = 0x100012e44a90005, negotiated timeout = 40000

But when I go in the gui, I see nothing. No clusters listed or anything. Is this expect behaviour, and if so, how can I pre-configure a cluster in kafka-manager to pass to the server?

[montana-ua]

I do it by ansible in the following way:

This is my defaults (any variable can be overriden by extra-vars)

kafka_cluster_config:
  name: "{{ kafka_cluster_name }}"
  zkHosts: "{{ kafka_cluster_zkhosts }}"
  kafkaVersion: "{{ kafka_cluster_version }}"
  jmxEnabled: "true"
  jmxUser: ""
  jmxPass: ""
  jmxSsl: "false"
  logkafkaEnabled: "false"
  pollConsumers: "true"
  filterConsumers: "true"
  activeOffsetCacheEnabled: "true"
  displaySizeEnabled: "false"
  tuning.brokerViewUpdatePeriodSeconds: 30
  tuning.clusterManagerThreadPoolSize: 10
  tuning.clusterManagerThreadPoolQueueSize: 100
  tuning.kafkaCommandThreadPoolSize: 10
  tuning.kafkaCommandThreadPoolQueueSize: 100
  tuning.logkafkaCommandThreadPoolSize: 10
  tuning.logkafkaCommandThreadPoolQueueSize: 100
  tuning.logkafkaUpdatePeriodSeconds: 30
  tuning.partitionOffsetCacheTimeoutSecs: 5
  tuning.brokerViewThreadPoolSize: 10
  tuning.brokerViewThreadPoolQueueSize: 1000
  tuning.offsetCacheThreadPoolSize: 10
  tuning.offsetCacheThreadPoolQueueSize: 1000
  tuning.kafkaAdminClientThreadPoolSize: 10
  tuning.kafkaAdminClientThreadPoolQueueSize: 1000
  securityProtocol: PLAINTEXT

and this is my task

# Add kafka cluster into kafka manager via POST API
###################################################
---

- block:
  
  - name: prepare data for the POST request
    set_fact:
      kafka_cluster_post_data: "{{ kafka_cluster_post_data | default('') }}&{{ item }}={{ kafka_cluster_config[item] }}"
    with_items: "{{ kafka_cluster_config }}"

  - name: add kafka cluster {{ kafka_cluster_name }}
    uri:
      url: "http://{{ first_host }}:{{ kafka_manager_port }}/clusters"
      method: POST
      body: "{{ kafka_cluster_post_data }}"
      status_code: 200

  run_once: true
  delegate_to: localhost

[montana-ua]

Also you can do it by znode module from ansible

- name: add kafka cluster {{ kafka_cluster_name }}
   znode:
     hosts: "{{ kafka_manager_zkhosts }}"
     name: "{{ kafka_manager_baseZkPath }}/configs/{{ kafka_cluster_name }}"
     value: "{{ kafka_cluster_data | to_json | string }}"
     state: present
   run_once: true
   delegate_to: 127.0.0.1