Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

The operator restarts periodically

See original GitHub issue

Describe the bug After deploy cluster operator restart periodically (typically after 15-20 minutes), without any error in logs

Environment:

  • Strimzi version: 0.29.0
  • Installation method: Helm chart
  • Kubernetes cluster: Kubernetes 1.22.2
  • Infrastructure: Kubadm
  • Kafka version: 3.1.0

YAML files and logs

Deploy using: helm upgrade kafka-operator strimzi/strimzi-kafka-operator --namespace kafka --version 0.29.0 --install --create-namespace --wait --timeout 300s --set resources.requests.cpu=50m

Operator last logs:

+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -XX:MinRAMPercentage=10 -XX:MaxRAMPercentage=20 -XX:InitialRAMPercentage=10 -classpath lib/io.strimzi.cluster-operator-0.29.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.12.2.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.1.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.strimzi.kafka-oauth-server-0.10.0.jar:lib/io.fabric8.kubernetes-model-common-5.12.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.12.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.12.6.jar:lib/io.fabric8.openshift-client-5.12.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machineconfig-5.12.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/io.strimzi.api-0.29.0.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.12.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.12.2.jar:lib/io.fabric8.openshift-model-hive-5.12.2.jar:lib/io.fabric8.kubernetes-model-apps-5.12.2.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.strimzi.kafka-oauth-server-plain-0.10.0.jar:lib/io.fabric8.openshift-model-operator-5.12.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.12.2.jar:lib/org.yaml.snakeyaml-1.27.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.strimzi.certificate-manager-0.29.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-core-5.12.2.jar:lib/io.strimzi.kafka-oauth-common-0.10.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.12.2.jar:lib/io.fabric8.kubernetes-model-scheduling-5.12.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-batch-5.12.2.jar:lib/io.fabric8.kubernetes-model-coordination-5.12.2.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.kubernetes-model-extensions-5.12.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-events-5.12.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/io.vertx.vertx-core-4.2.4.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.12.2.jar:lib/io.fabric8.openshift-model-whereabouts-5.12.2.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.strimzi.crd-annotations-0.29.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.12.2.jar:lib/io.fabric8.kubernetes-model-networking-5.12.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-monitoring-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-core-2.12.6.jar:lib/io.fabric8.kubernetes-client-5.12.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-model-certificates-5.12.2.jar:lib/io.fabric8.openshift-model-console-5.12.2.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.strimzi.kafka-oauth-client-0.10.0.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/org.apache.kafka.kafka-clients-3.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-installer-5.12.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-model-miscellaneous-5.12.2.jar:lib/io.fabric8.kubernetes-model-storageclass-5.12.2.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-node-5.12.2.jar:lib/io.strimzi.config-model-0.29.0.jar:lib/io.fabric8.openshift-model-machine-5.12.2.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.openshift-model-5.12.2.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.12.2.jar:lib/io.fabric8.kubernetes-model-rbac-5.12.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.operator-common-0.29.0.jar:lib/io.fabric8.openshift-model-tuned-5.12.2.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.12.6.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.netty.netty-common-4.1.77.Final.jar io.strimzi.operator.cluster.Main
2022-05-23 08:14:11 INFO  Main:60 - ClusterOperator 0.29.0 is starting
2022-05-23 08:14:13 INFO  Main:62 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[kafka],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000,createClusterRoles=false,networkPolicyGeneration=true,versions=versions{3.0.0={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0}, 3.0.1={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1}, 3.1.0={proto: 3.1 msg: 3.1 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0}, 3.1.1={proto: 3.1 msg: 3.1 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1}, 3.2.0={proto: 3.2 msg: 3.2 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0}},imagePullPolicy=null,imagePullSecrets=null,operatorNamespace=kafka,operatorNamespaceLabels=null,customResourceSelector=null,featureGates=FeatureGates(controlPlaneListener=true,ServiceAccountPatching=true,UseStrimziPodSets=false,UseKRaft=false),zkAdminSessionTimeoutMs=10000,dnsCacheTtlSec=30,podSetReconciliationOnly=false,podSetControllerWorkQueueSize=1024)
2022-05-23 08:14:16 WARN  PlatformFeaturesAvailability:157 - API Group route.openshift.io is not supported
2022-05-23 08:14:16 WARN  PlatformFeaturesAvailability:157 - API Group build.openshift.io is not supported
2022-05-23 08:14:16 WARN  PlatformFeaturesAvailability:157 - API Group image.openshift.io is not supported
2022-05-23 08:14:16 INFO  Main:85 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.22,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2022-05-23 08:14:16 INFO  Util:307 - Using config:
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	container: oci
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
	STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.29.0
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://10.111.195.194:9093
	JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny -XX:MinRAMPercentage=10 -XX:MaxRAMPercentage=20 -XX:InitialRAMPercentage=10
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.111.195.194
	DEFAULT_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
	DEFAULT_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://10.99.68.121:2181
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
	DEFAULT_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.111.195.194
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
	JAVA_MAIN: io.strimzi.operator.cluster.Main
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.111.195.194:9091
	STRIMZI_VERSION: 0.29.0
	STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.29.0
	STRIMZI_NAMESPACE: kafka
	STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.29.0
	TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
	STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.29.0
	STRIMZI_FEATURE_GATES: 
	DEFAULT_ZOOKEEPER_CLIENT_PORT: tcp://10.99.68.121:2181
	DEFAULT_ZOOKEEPER_CLIENT_SERVICE_HOST: 10.99.68.121
	TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
	KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
	DEFAULT_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTSTLS: 9093
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.111.195.194:9092
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	SHLVL: 0
	STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.21.5
	DEFAULT_ZOOKEEPER_CLIENT_SERVICE_PORT: 2181
	KUBERNETES_PORT: tcp://10.96.0.1:443
	STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	DEFAULT_ZOOKEEPER_CLIENT_SERVICE_PORT_TCP_CLIENTS: 2181
	JAVA_HOME: /usr/lib/jvm/jre-11
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.111.195.194
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	KUBERNETES_SERVICE_HOST: 10.96.0.1
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
	JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.29.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.12.2.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.1.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.strimzi.kafka-oauth-server-0.10.0.jar:lib/io.fabric8.kubernetes-model-common-5.12.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.12.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.12.6.jar:lib/io.fabric8.openshift-client-5.12.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machineconfig-5.12.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/io.strimzi.api-0.29.0.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.12.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.12.2.jar:lib/io.fabric8.openshift-model-hive-5.12.2.jar:lib/io.fabric8.kubernetes-model-apps-5.12.2.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.strimzi.kafka-oauth-server-plain-0.10.0.jar:lib/io.fabric8.openshift-model-operator-5.12.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.12.2.jar:lib/org.yaml.snakeyaml-1.27.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.strimzi.certificate-manager-0.29.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-core-5.12.2.jar:lib/io.strimzi.kafka-oauth-common-0.10.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.12.2.jar:lib/io.fabric8.kubernetes-model-scheduling-5.12.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-batch-5.12.2.jar:lib/io.fabric8.kubernetes-model-coordination-5.12.2.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.kubernetes-model-extensions-5.12.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-events-5.12.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/io.vertx.vertx-core-4.2.4.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.12.2.jar:lib/io.fabric8.openshift-model-whereabouts-5.12.2.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.strimzi.crd-annotations-0.29.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.12.2.jar:lib/io.fabric8.kubernetes-model-networking-5.12.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-monitoring-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-core-2.12.6.jar:lib/io.fabric8.kubernetes-client-5.12.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-model-certificates-5.12.2.jar:lib/io.fabric8.openshift-model-console-5.12.2.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.strimzi.kafka-oauth-client-0.10.0.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/org.apache.kafka.kafka-clients-3.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-installer-5.12.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-model-miscellaneous-5.12.2.jar:lib/io.fabric8.kubernetes-model-storageclass-5.12.2.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-node-5.12.2.jar:lib/io.strimzi.config-model-0.29.0.jar:lib/io.fabric8.openshift-model-machine-5.12.2.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.openshift-model-5.12.2.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.12.2.jar:lib/io.fabric8.kubernetes-model-rbac-5.12.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.operator-common-0.29.0.jar:lib/io.fabric8.openshift-model-tuned-5.12.2.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.12.6.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.netty.netty-common-4.1.77.Final.jar
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
	STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	TINI_VERSION: v0.19.0
	DEFAULT_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
	STRIMZI_OPERATION_TIMEOUT_MS: 300000
	DEFAULT_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
	KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
	DEFAULT_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
	STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.29.0
	DEFAULT_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 10.111.195.194
	KUBERNETES_SERVICE_PORT: 443
	STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
	DEFAULT_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
	STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.29.0
	TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
	HOSTNAME: strimzi-cluster-operator-56b86c4f59-6kfsp
	DEFAULT_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 10.99.68.121
	STRIMZI_KAFKA_CONNECT_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	STRIMZI_KAFKA_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	KUBERNETES_PORT_443_TCP_PORT: 443
	DEFAULT_KAFKA_BOOTSTRAP_PORT: tcp://10.111.195.194:9091
	STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	HOME: /home/strimzi
	MALLOC_ARENA_MAX: 2
	STRIMZI_OPERATOR_NAMESPACE: kafka

2022-05-23 08:14:16 INFO  ClusterOperator:79 - Creating ClusterOperator for namespace kafka
2022-05-23 08:14:16 INFO  ClusterOperator:94 - Starting ClusterOperator for namespace kafka
2022-05-23 08:14:17 INFO  ClusterOperator:113 - Opened watch for Kafka operator
2022-05-23 08:14:17 INFO  ClusterOperator:113 - Opened watch for KafkaMirrorMaker operator
2022-05-23 08:14:17 INFO  ClusterOperator:113 - Opened watch for KafkaConnect operator
2022-05-23 08:14:17 INFO  ClusterOperator:113 - Opened watch for KafkaBridge operator
2022-05-23 08:14:17 INFO  ClusterOperator:113 - Opened watch for KafkaMirrorMaker2 operator
2022-05-23 08:14:17 INFO  ClusterOperator:125 - Setting up periodic reconciliation for namespace kafka
2022-05-23 08:14:18 INFO  ClusterOperator:192 - ClusterOperator is now ready (health server listening on 8080)
2022-05-23 08:14:18 INFO  Main:155 - Cluster Operator verticle started in namespace kafka without label selector
2022-05-23 08:14:19 INFO  OperatorWatcher:38 - Reconciliation #1(watch) Kafka(kafka/default): Kafka default in namespace kafka was ADDED
2022-05-23 08:14:19 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2022-05-23 08:14:19 INFO  AbstractOperator:226 - Reconciliation #1(watch) Kafka(kafka/default): Kafka default will be checked for creation or modification
2022-05-23 08:14:20 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2022-05-23 08:14:31 INFO  AbstractOperator:517 - Reconciliation #1(watch) Kafka(kafka/default): reconciled
2022-05-23 08:16:17 INFO  ClusterOperator:128 - Triggering periodic reconciliation for namespace kafka
2022-05-23 08:16:18 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
2022-05-23 08:16:18 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2022-05-23 08:16:18 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2022-05-23 08:16:18 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
2022-05-23 08:16:18 INFO  AbstractOperator:226 - Reconciliation #2(timer) Kafka(kafka/default): Kafka default will be checked for creation or modification
2022-05-23 08:16:18 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'
2022-05-23 08:16:18 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnectors' with unstable version 'v1beta2'
2022-05-23 08:16:24 INFO  AbstractOperator:517 - Reconciliation #2(timer) Kafka(kafka/default): reconciled

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
    meta.helm.sh/release-name: kafka-operator
    meta.helm.sh/release-namespace: kafka
  creationTimestamp: "2022-04-27T10:52:31Z"
  generation: 3
  labels:
    app: strimzi
    app.kubernetes.io/managed-by: Helm
    chart: strimzi-kafka-operator-0.29.0
    component: deployment
    heritage: Helm
    release: kafka-operator
  name: strimzi-cluster-operator
  namespace: kafka
  resourceVersion: "49586023"
  uid: 75a431c1-f0f7-4626-9dc3-8eaa94280485
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: strimzi-cluster-operator
      strimzi.io/kind: cluster-operator
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-05-11T11:05:13+02:00"
      creationTimestamp: null
      labels:
        name: strimzi-cluster-operator
        strimzi.io/kind: cluster-operator
    spec:
      containers:
      - args:
        - /opt/strimzi/bin/cluster_operator_run.sh
        env:
        - name: STRIMZI_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
          value: "120000"
        - name: STRIMZI_OPERATION_TIMEOUT_MS
          value: "300000"
        - name: STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE
          value: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
        - name: STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE
          value: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
        - name: STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE
          value: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
        - name: STRIMZI_KAFKA_IMAGES
          value: |
            3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
            3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
            3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
            3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
            3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
        - name: STRIMZI_KAFKA_CONNECT_IMAGES
          value: |
            3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
            3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
            3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
            3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
            3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
        - name: STRIMZI_KAFKA_MIRROR_MAKER_IMAGES
          value: |
            3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
            3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
            3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
            3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
            3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
        - name: STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES
          value: |
            3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
            3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
            3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
            3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
            3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
        - name: STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE
          value: quay.io/strimzi/operator:0.29.0
        - name: STRIMZI_DEFAULT_USER_OPERATOR_IMAGE
          value: quay.io/strimzi/operator:0.29.0
        - name: STRIMZI_DEFAULT_KAFKA_INIT_IMAGE
          value: quay.io/strimzi/operator:0.29.0
        - name: STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE
          value: quay.io/strimzi/kafka-bridge:0.21.5
        - name: STRIMZI_DEFAULT_JMXTRANS_IMAGE
          value: quay.io/strimzi/jmxtrans:0.29.0
        - name: STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE
          value: quay.io/strimzi/kaniko-executor:0.29.0
        - name: STRIMZI_DEFAULT_MAVEN_BUILDER
          value: quay.io/strimzi/maven-builder:0.29.0
        - name: STRIMZI_OPERATOR_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: STRIMZI_FEATURE_GATES
        image: quay.io/strimzi/operator:0.29.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthy
            port: http
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 1
        name: strimzi-cluster-operator
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: http
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 384Mi
          requests:
            cpu: 50m
            memory: 384Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: strimzi-tmp
        - mountPath: /opt/strimzi/custom-config/
          name: co-config-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: strimzi-cluster-operator
      serviceAccountName: strimzi-cluster-operator
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
          sizeLimit: 1Mi
        name: strimzi-tmp
      - configMap:
          defaultMode: 420
          name: strimzi-cluster-operator
        name: co-config-volume
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-04-27T10:52:31Z"
    lastUpdateTime: "2022-05-21T12:40:45Z"
    message: ReplicaSet "strimzi-cluster-operator-56b86c4f59" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2022-05-23T08:14:45Z"
    lastUpdateTime: "2022-05-23T08:14:45Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

  strimzi-cluster-operator:
    Container ID:  containerd://854ce1194ff460094118359bcc9fa8b3f08431f54f4fc5aee06dead2bbdb8ac0
    Image:         quay.io/strimzi/operator:0.29.0
    Image ID:      quay.io/strimzi/operator@sha256:afde94c6d2544207cb7f3c94b54455643558a9d5daaa13b9f736cc8da20e319c
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      /opt/strimzi/bin/cluster_operator_run.sh
    State:          Running
      Started:      Mon, 23 May 2022 10:40:45 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 23 May 2022 10:22:35 +0200
      Finished:     Mon, 23 May 2022 10:40:45 +0200
    Ready:          True
    Restart Count:  162
    Limits:
      cpu:     1
      memory:  384Mi
    Requests:
      cpu:      50m
      memory:   384Mi
    Liveness:   http-get http://:http/healthy delay=10s timeout=1s period=30s #success=1 #failure=3
    Readiness:  http-get http://:http/ready delay=10s timeout=1s period=30s #success=1 #failure=3
    Environment:
      STRIMZI_NAMESPACE:                                  kafka (v1:metadata.namespace)
      STRIMZI_FULL_RECONCILIATION_INTERVAL_MS:            120000
      STRIMZI_OPERATION_TIMEOUT_MS:                       300000
      STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE:  quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
      STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE:               quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
      STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE:               quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
      STRIMZI_KAFKA_IMAGES:                               3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
                                                          3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
                                                          3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
                                                          3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
                                                          3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
                                                          
      STRIMZI_KAFKA_CONNECT_IMAGES:                       3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
                                                          3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
                                                          3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
                                                          3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
                                                          3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
                                                          
      STRIMZI_KAFKA_MIRROR_MAKER_IMAGES:                  3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
                                                          3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
                                                          3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
                                                          3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
                                                          3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
                                                          
      STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES:                3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
                                                          3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
                                                          3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
                                                          3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
                                                          3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
                                                          
      STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE:               quay.io/strimzi/operator:0.29.0
      STRIMZI_DEFAULT_USER_OPERATOR_IMAGE:                quay.io/strimzi/operator:0.29.0
      STRIMZI_DEFAULT_KAFKA_INIT_IMAGE:                   quay.io/strimzi/operator:0.29.0
      STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE:                 quay.io/strimzi/kafka-bridge:0.21.5
      STRIMZI_DEFAULT_JMXTRANS_IMAGE:                     quay.io/strimzi/jmxtrans:0.29.0
      STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE:              quay.io/strimzi/kaniko-executor:0.29.0
      STRIMZI_DEFAULT_MAVEN_BUILDER:                      quay.io/strimzi/maven-builder:0.29.0
      STRIMZI_OPERATOR_NAMESPACE:                         kafka (v1:metadata.namespace)
      STRIMZI_FEATURE_GATES:                              
    Mounts:
      /opt/strimzi/custom-config/ from co-config-volume (rw)
      /tmp from strimzi-tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9hp82 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  strimzi-tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  1Mi
  co-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      strimzi-cluster-operator
    Optional:  false
  kube-api-access-9hp82:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Warning  BackOff  25m (x2 over 20h)      kubelet  Back-off restarting failed container
  Normal   Created  7m19s (x163 over 44h)  kubelet  Created container strimzi-cluster-operator
  Normal   Started  7m19s (x163 over 44h)  kubelet  Started container strimzi-cluster-operator
  Normal   Pulled   7m19s (x162 over 43h)  kubelet  Container image "quay.io/strimzi/operator:0.29.0" already present on machine

default-entity-operator-789c4c944-khnqs     3/3     Running   0                 43h
default-kafka-0                             1/1     Running   0                 44h
default-zookeeper-0                         1/1     Running   0                 44h
strimzi-cluster-operator-56b86c4f59-6kfsp   1/1     Running   162 (6m19s ago)   44h

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:16 (8 by maintainers)

github_iconTop GitHub Comments

3reactions
angus1718commented, Jun 8, 2022

I am using cgroup v2 and seeing the same issue on strimzi 0.29. i found that oom killed occurred and the operator restarted. i search for such a reason and find this article it seems jdk11 doesn’t support cgroup v2 so *RAMPercentage uses host memory, not container memory

1reaction
AydinChavezcommented, Jun 13, 2022

You’re welcome! FYI I’ve updated the PR in order to do the same but with less logic after applying suggestions from @scholzj here the last version of it launch_java.sh

Am also affected - for applying the mentioned fixes in launch_java.sh - I guess I do need to rebuild the strimzi-operator image with the patched launch_java.sh, right? Will this fix be shipped soon with a release, so I can skip this step as a workaround?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Pods restarted regularly every 10 hours - IBM
The ibm-cert-manager-operator pod automatically restarts pods whenever the Certificate they use is renewed. However, in foundational services 3.13, ...
Read more >
Troubleshooting Operator issues - OpenShift Documentation
Disabling the Machine Config Operator from automatically rebooting by using the CLI ... To avoid unwanted disruptions from changes made by the Machine...
Read more >
Many infra pods restarting periodically on master node
Description of problem: Some infra pods always restarted periodically on master node after cluster is running for a while. More details see the...
Read more >
Automatically elastic operator pod went down and up
Deployed Elasticsearch cluster in GCP. Without making any changes, operator pod goes down and after sometime pods get restarted.
Read more >
16.10 - Restart Log Tables - Parallel Transporter
The information in the restart log table is normally not accessed directly by Teradata PT users, but is automatically used by the Teradata...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Operator is leaking files in /tmp, running out of disk space

Next page

mirromaker2 can not reconcile - NullPointerException