Kafka-Connect Readiness and Liveness failed error

Hello Everyone, I am trying to deploy kafka-connect on strimzi operator but getting Readiness and Liveness failed error.looked for this in issues,but did not found any working solution.

Strimzi operator version : 0.19.0

Kafka version : 2.5.0

strimzi/kafka-connect:0.11.4-kafka-2.1.0

Kafka is also running in the same namespace with following bootstrap data-kafka-bootstrap ClusterIP 172.20.203.132 <none> 9091/TCP,9092/TCP,9093/TCP,9404/TCP 24d

Following is my kafka-connect config.I have not configured any readiness or liveness in this config. With Readiness and Liveness probe Disabled

versionNumber: 2.5.0
replicaCount: 1
namespace: kafka-cluster
bootstrapServersName: "data-kafka-bootstrap:9092"

image:
  repository: strimzi/kafka-connect
  tag: 0.11.4-kafka-2.1.0
  pullPolicy: Always

tls:
  certname : data-cluster-ca-cert

imagePullSecrets: []
nameOverride: "data"
fullnameOverride: ""

config_storage_replication_factor: 3
offset_storage_replication_factor: 3
status_storage_replication_factor: 3

Here is the describe output

4s         Normal    Scheduled           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Successfully assigned kafka-cluster/data-kafka-connect-cluster-connect-74d98646d8-d825l to ip-10-0-1-15.us-east-2.compute.internal
13s         Normal    Pulling             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Pulling image "strimzi/kafka-connect:0.11.4-kafka-2.1.0"
12s         Normal    Pulled              pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Successfully pulled image "strimzi/kafka-connect:0.11.4-kafka-2.1.0"
12s         Normal    Created             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Created container data-kafka-connect-cluster-connect
11s         Normal    Started             pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Started container data-kafka-connect-cluster-connect
14s         Normal    SuccessfulCreate    replicaset/data-kafka-connect-cluster-connect-74d98646d8   Created pod: data-kafka-connect-cluster-connect-74d98646d8-d825l
14s         Normal    NoPods              poddisruptionbudget/data-kafka-connect-cluster-connect     No matching pods found
14s         Normal    ScalingReplicaSet   deployment/data-kafka-connect-cluster-connect              Scaled up replica set data-kafka-connect-cluster-connect-74d98646d8 to 1
6m59s       Warning   BackOff             pod/kafka-connect-cluster-connect-596b456dbf-5s2qq         Back-off restarting failed container
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Readiness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Readiness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused
0s          Warning   Unhealthy           pod/data-kafka-connect-cluster-connect-74d98646d8-d825l    Liveness probe failed: Get http://10.0.1.51:8083/: dial tcp 10.0.1.51:8083: connect: connection refused

Although strimzi Operator has by default this configured.

  livenessProbe:
    initialDelaySeconds: 10
    periodSeconds: 30
  readinessProbe:
    initialDelaySeconds: 10
    periodSeconds: 30

I have also tried configuring kafka-connect with Readiness and Liveness probe with following config.But still same result.

With Readiness and Liveness probe Enabled

    livenessProbe:
      httpGet:
          path: /connectors
          port: 8083
      initialDelaySeconds: 60
      periodSeconds: 5
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 10
    readinessProbe:
      httpGet:
          path: /connectors
          port: 8083
      initialDelaySeconds: 40
      periodSeconds: 10
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 3

Please let me know if anything else required.

Thanks in advance.

[scholzj]

Well, it is your choice really (consider security, versus performance etc.) - I would normally recommend using TLS unless you have some actual reasons against it.

But in any case, you can:

• Keep using the port 9092 and jut disable the TLS
• Keep the TLS but use port 9093 in the bootstrap address which is used in the Kafka cluster for TLS

Both should work.

[prash911]

Noted.

Thanks a lot for your help.

I am able to use both Tls and non tls ports with right config.