Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Strimzi recovery failure with Velero
See original GitHub issueAfter restore using velero
Below details …
[root@sync-cluster setup-files]# oc get po
NAME READY STATUS RESTARTS AGE
akhq-ddcfcfd8b-5ftqn 1/1 Running 0 5m10s
kafka-acl-viewer-f79b7fff6-gzgdw 1/1 Running 4 5m10s
strimzi-cluster-operator-68b6d59f74-4pz7c 0/1 CrashLoopBackOff 5 5m10s
sync-bridge-bridge-78bf5cf75f-k2vvh 1/1 Running 1 5m10s
sync-cluster-entity-operator-59b97b8c88-zlfrw 1/3 CrashLoopBackOff 11 5m10s
sync-cluster-kafka-0 2/2 Running 1 5m9s
sync-cluster-kafka-1 2/2 Running 1 5m9s
sync-cluster-kafka-2 2/2 Running 1 5m9s
sync-cluster-zookeeper-0 1/1 Running 0 5m9s
sync-cluster-zookeeper-1 1/1 Running 0 5m9s
sync-cluster-zookeeper-2 1/1 Running 0 5m9s
[root@sync-cluster setup-files]# oc logs -f strimzi-cluster-operator-68b6d59f74-4pz7c
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
++ java -version
++ sed -E -n 's/.* version "([0-9]*).*$/\1/p'
+ JAVA_MAJOR_VERSION=11
+ '[' 11 -ge 9 ']'
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.kafka-oauth-client-0.5.0.jar:lib/io.strimzi.api-0.19.0.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.keycloak.keycloak-core-10.0.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/io.strimzi.config-model-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.keycloak.keycloak-common-10.0.0.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-common-0.5.0.jar:lib/io.strimzi.kafka-oauth-server-0.5.0.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.62.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar io.strimzi.operator.cluster.Main
2020-10-04 02:25:30 INFO Main:60 - ClusterOperator 0.19.0 is starting
2020-10-04 02:25:33 INFO Main:86 - Environment facts gathered: ClusterOperatorConfig(KubernetesVersion=1.19,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false,OpenShiftDeploymentConfigs=false)
2020-10-04 02:25:33 INFO Util:268 - Using config:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SYNC_CLUSTER_KAFKA_2_SERVICE_PORT: 9094
STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: strimzi/operator:0.19.0
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP: tcp://10.96.164.152:9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP: tcp://10.96.142.104:9094
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT: tcp://10.96.164.152:9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
KAFKA_ACL_VIEWER_SERVICE_HOST: 10.96.208.142
STRIMZI_DEFAULT_TLS_SIDECAR_KAFKA_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT: tcp://10.96.98.125:9091
STRIMZI_HOME: /opt/strimzi
PWD: /opt/strimzi
SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT_REST_API: 8080
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP: tcp://10.96.38.96:8080
JAVA_MAIN: io.strimzi.operator.cluster.Main
AKHQ_PORT_80_TCP_PORT: 80
STRIMZI_DEFAULT_JMXTRANS_IMAGE: strimzi/jmxtrans:0.19.0
STRIMZI_NAMESPACE: streaming
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_HOST: 10.96.164.152
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PROTO: tcp
KAFKA_ACL_VIEWER_PORT: tcp://10.96.208.142:80
KAFKA_ACL_VIEWER_PORT_80_TCP: tcp://10.96.208.142:80
STRIMZI_LOG_LEVEL: INFO
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
SYNC_CLUSTER_KAFKA_1_SERVICE_PORT: 9094
KUBERNETES_SERVICE_PORT_HTTPS: 443
AKHQ_PORT: tcp://10.96.4.192:80
SHLVL: 0
AKHQ_PORT_80_TCP: tcp://10.96.4.192:80
AKHQ_SERVICE_PORT: 80
KAFKA_ACL_VIEWER_PORT_80_TCP_ADDR: 10.96.208.142
TINI_SHA256: 12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855
AKHQ_SERVICE_PORT_HTTP: 80
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP: tcp://10.96.151.227:9094
STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0
KUBERNETES_PORT: tcp://10.96.0.1:443
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 10.96.98.125
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.kafka-oauth-client-0.5.0.jar:lib/io.strimzi.api-0.19.0.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.keycloak.keycloak-core-10.0.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/io.strimzi.config-model-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/org.keycloak.keycloak-common-10.0.0.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-common-0.5.0.jar:lib/io.strimzi.kafka-oauth-server-0.5.0.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.62.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar
SYNC_CLUSTER_KAFKA_2_SERVICE_HOST: 10.96.151.227
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://10.96.16.2:2181
SYNC_CLUSTER_KAFKA_0_SERVICE_HOST: 10.96.142.104
STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.96.98.125:9092
KAFKA_ACL_VIEWER_PORT_80_TCP_PROTO: tcp
SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_HOST: 10.96.16.2
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_ADDR: 10.96.151.227
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PORT: 8080
KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_ADDR: 10.96.100.188
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_ADDR: 10.96.142.104
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP: tcp://10.96.100.188:9094
STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0
KUBERNETES_PORT_443_TCP_PROTO: tcp
STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: strimzi/operator:0.19.0
STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: strimzi/operator:0.19.0
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
STRIMZI_KAFKA_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0
SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT: 2181
STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_ADDR: 10.96.164.152
SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT: 8080
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PROTO: tcp
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
SYNC_CLUSTER_KAFKA_2_PORT: tcp://10.96.151.227:9094
SYNC_CLUSTER_KAFKA_1_PORT: tcp://10.96.100.188:9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.96.98.125
SYNC_CLUSTER_KAFKA_1_SERVICE_HOST: 10.96.100.188
KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT: tcp://10.96.16.2:2181
STRIMZI_VERSION: 0.19.0
SYNC_CLUSTER_KAFKA_0_PORT: tcp://10.96.142.104:9094
AKHQ_PORT_80_TCP_PROTO: tcp
AKHQ_SERVICE_HOST: 10.96.4.192
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT: 9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTSTLS: 9093
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PROTO: tcp
AKHQ_PORT_80_TCP_ADDR: 10.96.4.192
SYNC_CLUSTER_KAFKA_0_SERVICE_PORT: 9094
STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: strimzi/kafka-bridge:0.18.0
SYNC_BRIDGE_BRIDGE_SERVICE_PORT: tcp://10.96.38.96:8080
SYNC_CLUSTER_KAFKA_0_SERVICE_PORT_TCP_EXTERNAL: 9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.96.98.125
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.96.0.1
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://10.96.98.125:9093
SYNC_CLUSTER_KAFKA_1_SERVICE_PORT_TCP_EXTERNAL: 9094
STRIMZI_KAFKA_CONNECT_S2I_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PORT: 9094
SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_HOST: 10.96.38.96
TINI_VERSION: v0.18.0
STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
KAFKA_ACL_VIEWER_SERVICE_PORT_80_TCP: 80
STRIMZI_OPERATION_TIMEOUT_MS: 300000
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.96.98.125
KAFKA_ACL_VIEWER_SERVICE_PORT: 80
SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT_TCP_CLIENTS: 2181
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PORT: 9094
KUBERNETES_SERVICE_PORT: 443
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.96.98.125:9091
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PORT: 9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PORT: 9094
STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: strimzi/kafka:0.19.0-kafka-2.5.0
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 10.96.16.2
HOSTNAME: strimzi-cluster-operator-68b6d59f74-4pz7c
STRIMZI_KAFKA_CONNECT_IMAGES: 2.4.0=strimzi/kafka:0.19.0-kafka-2.4.0
2.4.1=strimzi/kafka:0.19.0-kafka-2.4.1
2.5.0=strimzi/kafka:0.19.0-kafka-2.5.0
KUBERNETES_PORT_443_TCP_PORT: 443
KAFKA_ACL_VIEWER_PORT_80_TCP_PORT: 80
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_ADDR: 10.96.38.96
SYNC_CLUSTER_KAFKA_2_SERVICE_PORT_TCP_EXTERNAL: 9094
HOME: /home/strimzi
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT_TCP_EXTERNAL: 9094
MALLOC_ARENA_MAX: 2
2020-10-04 02:25:33 INFO Main:127 - The KafkaConnectS2I custom resource definition can only be used in environment which supports OpenShift build, image and apps APIs. These APIs do not seem to be supported in this environment.
2020-10-04 02:25:33 INFO ClusterOperator:81 - Creating ClusterOperator for namespace streaming
2020-10-04 02:25:33 INFO ClusterOperator:98 - Starting ClusterOperator for namespace streaming
2020-10-04 02:25:34 WARN WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkas.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkas" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:34 INFO AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:34 WARN WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkamirrormakers.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkamirrormakers" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:34 INFO AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:35 WARN WatchConnectionManager:366 - Executor didn't terminate in time after shutdown in close(), killing it in: io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@59c09998
2020-10-04 02:25:35 WARN WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkaconnects.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkaconnects" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:35 INFO AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:35 WARN WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkabridges.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkabridges" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:35 INFO AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:36 WARN WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkamirrormaker2s.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkamirrormaker2s" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:36 INFO AbstractOperator:304 - Watcher closed in namespace streaming
2020-10-04 02:25:36 WARN WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkaconnectors.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkaconnectors" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:36 WARN WatchConnectionManager:202 - Exec Failure: HTTP 403, Status: 403 - kafkarebalances.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkarebalances" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-04 02:25:36 ERROR Main:162 - Cluster Operator verticle in namespace streaming failed to start
io.fabric8.kubernetes.client.KubernetesClientException: kafkas.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:strimzi-cluster-operator" cannot watch resource "kafkas" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:206) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Suppressed: java.lang.Throwable: waiting here
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:135) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:354) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:759) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:738) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:69) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:63) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.strimzi.operator.common.AbstractOperator.lambda$createWatch$9(AbstractOperator.java:293) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.strimzi.operator.common.Util.lambda$async$0(Util.java:49) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[root@sync-cluster setup-files]# oc describe po sync-cluster-entity-operator-59b97b8c88-zlfrw
Name: sync-cluster-entity-operator-59b97b8c88-zlfrw
Namespace: streaming
Priority: 0
Node: kind-control-plane/172.17.0.2
Start Time: Sun, 04 Oct 2020 02:21:43 +0000
Labels: app.kubernetes.io/instance=sync-cluster
app.kubernetes.io/managed-by=strimzi-cluster-operator
app.kubernetes.io/name=entity-operator
app.kubernetes.io/part-of=strimzi-sync-cluster
pod-template-hash=59b97b8c88
strimzi.io/cluster=sync-cluster
strimzi.io/kind=Kafka
strimzi.io/name=sync-cluster-entity-operator
velero.io/backup-name=velero-bkp-sync-cluster.streaming.full-04-10-2020-02-14-43
velero.io/restore-name=velero-restore-sync-cluster-04-10-2020-02-21-40
Annotations: strimzi.io/clients-ca-cert-generation: 0
strimzi.io/cluster-ca-cert-generation: 0
Status: Running
IP: 10.244.0.27
IPs:
IP: 10.244.0.27
Controlled By: ReplicaSet/sync-cluster-entity-operator-59b97b8c88
Containers:
topic-operator:
Container ID: containerd://b51a51f9a72c2c3c8db61cf9e8aa96ad235880688b0682076a768c4de3bf49d5
Image: strimzi/operator:0.19.0
Image ID: docker.io/strimzi/operator@sha256:d8fed9eb28c6d024670b060527681e21c6744a6c7dd2bf48450ce577382d3646
Port: 8080/TCP
Host Port: 0/TCP
Args:
/opt/strimzi/bin/topic_operator_run.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 04 Oct 2020 02:29:23 +0000
Finished: Sun, 04 Oct 2020 02:29:54 +0000
Ready: False
Restart Count: 7
Liveness: http-get http://:healthcheck/healthy delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:healthcheck/ready delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=sync-cluster
STRIMZI_KAFKA_BOOTSTRAP_SERVERS: sync-cluster-kafka-bootstrap:9091
STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
STRIMZI_NAMESPACE: streaming
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 90000
STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 20000
STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
STRIMZI_TLS_ENABLED: true
STRIMZI_GC_LOG_ENABLED: false
Mounts:
/etc/tls-sidecar/cluster-ca-certs/ from cluster-ca-certs (rw)
/etc/tls-sidecar/eo-certs/ from eo-certs (rw)
/opt/topic-operator/custom-config/ from entity-topic-operator-metrics-and-logging (rw)
/var/run/secrets/kubernetes.io/serviceaccount from sync-cluster-entity-operator-token-s75jc (ro)
user-operator:
Container ID: containerd://dcdbe68ba4c33004d3cfa8b0ccdf6861df3cf2d8f51a65fed34b5976b9047d6e
Image: strimzi/operator:0.19.0
Image ID: docker.io/strimzi/operator@sha256:d8fed9eb28c6d024670b060527681e21c6744a6c7dd2bf48450ce577382d3646
Port: 8081/TCP
Host Port: 0/TCP
Args:
/opt/strimzi/bin/user_operator_run.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 04 Oct 2020 02:27:57 +0000
Finished: Sun, 04 Oct 2020 02:27:59 +0000
Ready: False
Restart Count: 6
Liveness: http-get http://:healthcheck/healthy delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:healthcheck/ready delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
STRIMZI_KAFKA_BOOTSTRAP_SERVERS: sync-cluster-kafka-bootstrap:9091
STRIMZI_NAMESPACE: streaming
STRIMZI_LABELS: strimzi.io/cluster=sync-cluster
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 6000
STRIMZI_CA_KEY_NAME: sync-cluster-clients-ca
STRIMZI_CA_CERT_NAME: sync-cluster-clients-ca-cert
STRIMZI_CA_NAMESPACE: streaming
STRIMZI_CA_VALIDITY: 365
STRIMZI_CA_RENEWAL: 30
STRIMZI_CLUSTER_CA_CERT_SECRET_NAME: sync-cluster-cluster-ca-cert
STRIMZI_EO_KEY_SECRET_NAME: sync-cluster-entity-operator-certs
STRIMZI_GC_LOG_ENABLED: false
Mounts:
/etc/tls-sidecar/cluster-ca-certs/ from cluster-ca-certs (rw)
/etc/tls-sidecar/eo-certs/ from eo-certs (rw)
/opt/user-operator/custom-config/ from entity-user-operator-metrics-and-logging (rw)
/var/run/secrets/kubernetes.io/serviceaccount from sync-cluster-entity-operator-token-s75jc (ro)
tls-sidecar:
Container ID: containerd://9619d9e30ad1c5c87f94ee94862a41025f19e71890a1a52b0189d05ba2076401
Image: strimzi/kafka:0.19.0-kafka-2.5.0
Image ID: docker.io/strimzi/kafka@sha256:961419d795036f5f491817a5652c8f30c531a7b3cafd245d9473812a6a632c19
Port: <none>
Host Port: <none>
Command:
/opt/stunnel/entity_operator_stunnel_run.sh
State: Running
Started: Sun, 04 Oct 2020 02:21:46 +0000
Ready: True
Restart Count: 0
Liveness: exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3
Environment:
TLS_SIDECAR_LOG_LEVEL: notice
STRIMZI_ZOOKEEPER_CONNECT: sync-cluster-zookeeper-client:2181
Mounts:
/etc/tls-sidecar/cluster-ca-certs/ from cluster-ca-certs (rw)
/etc/tls-sidecar/eo-certs/ from eo-certs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from sync-cluster-entity-operator-token-s75jc (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
entity-topic-operator-metrics-and-logging:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sync-cluster-entity-topic-operator-config
Optional: false
entity-user-operator-metrics-and-logging:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sync-cluster-entity-user-operator-config
Optional: false
eo-certs:
Type: Secret (a volume populated by a Secret)
SecretName: sync-cluster-entity-operator-certs
Optional: false
cluster-ca-certs:
Type: Secret (a volume populated by a Secret)
SecretName: sync-cluster-cluster-ca-cert
Optional: false
sync-cluster-entity-operator-token-s75jc:
Type: Secret (a volume populated by a Secret)
SecretName: sync-cluster-entity-operator-token-s75jc
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m55s default-scheduler Successfully assigned streaming/sync-cluster-entity-operator-59b97b8c88-zlfrw to kind-control-plane
Normal Created 9m53s kubelet Created container topic-operator
Normal Started 9m53s kubelet Started container topic-operator
Normal Started 9m52s kubelet Started container tls-sidecar
Normal Created 9m52s kubelet Created container tls-sidecar
Normal Pulled 9m52s kubelet Container image "strimzi/kafka:0.19.0-kafka-2.5.0" already present on machine
Normal Started 9m20s (x3 over 9m52s) kubelet Started container user-operator
Normal Created 9m20s (x3 over 9m53s) kubelet Created container user-operator
Normal Pulled 9m20s (x3 over 9m53s) kubelet Container image "strimzi/operator:0.19.0" already present on machine
Warning Unhealthy 9m16s (x3 over 9m36s) kubelet Readiness probe failed: Get "http://10.244.0.27:8080/ready": dial tcp 10.244.0.27:8080: connect: connection refused
Normal Pulled 9m14s (x2 over 9m53s) kubelet Container image "strimzi/operator:0.19.0" already present on machine
Warning Unhealthy 9m14s (x3 over 9m34s) kubelet Liveness probe failed: Get "http://10.244.0.27:8080/healthy": dial tcp 10.244.0.27:8080: connect: connection refused
Normal Killing 9m14s kubelet Container topic-operator failed liveness probe, will be restarted
Warning BackOff 4m49s (x36 over 9m35s) kubelet Back-off restarting failed container
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]# oc logs -f sync-cluster-entity-operator-59b97b8c88-zlfrw -c topic-operator
Preparing certificates for internal communication
Adding /etc/tls-sidecar/cluster-ca-certs/ca.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias ca
Certificate was added to keystore
Preparing certificates for internal communication is complete
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' false == true ']'
++ echo ''
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
++ java -version
++ sed -E -n 's/.* version "([0-9]*).*$/\1/p'
+ JAVA_MAJOR_VERSION=11
+ '[' 11 -ge 9 ']'
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.topic-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar io.strimzi.operator.topic.Main
[2020-10-04 02:29:24,965] INFO <Main :31> [main ] TopicOperator 0.19.0 is starting
[2020-10-04 02:29:25,597] INFO <Session :66> [main ] Using config:
STRIMZI_TRUSTSTORE_LOCATION: /tmp/topic-operator/replication.truststore.p12
STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=sync-cluster
STRIMZI_KAFKA_BOOTSTRAP_SERVERS: sync-cluster-kafka-bootstrap:9091
STRIMZI_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS
STRIMZI_NAMESPACE: streaming
STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 20000
STRIMZI_TOPICS_PATH: /strimzi/topics
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 90000
STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
STRIMZI_TLS_ENABLED: true
STRIMZI_KEYSTORE_PASSWORD: ********
STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
STRIMZI_KEYSTORE_LOCATION: /tmp/topic-operator/replication.keystore.p12
TC_ZK_CONNECTION_TIMEOUT_MS: 20000
STRIMZI_TRUSTSTORE_PASSWORD: ********
STRIMZI_REASSIGN_THROTTLE: 9223372036854775807
[2020-10-04 02:29:25,619] INFO <Session :135> [oop-thread-0] Starting
[2020-10-04 02:29:25,646] INFO <ClientConfig:347> [oop-thread-0] AdminClientConfig values:
bootstrap.servers = [sync-cluster-kafka-bootstrap:9091]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = HTTPS
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = /tmp/topic-operator/replication.keystore.p12
ssl.keystore.password = [hidden]
ssl.keystore.type = JKS
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = /tmp/topic-operator/replication.truststore.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
[2020-10-04 02:29:25,868] WARN <ClientConfig:355> [oop-thread-0] The configuration 'ssl.truststore.location' was supplied but isn't a known config.
[2020-10-04 02:29:25,868] WARN <ClientConfig:355> [oop-thread-0] The configuration 'ssl.keystore.password' was supplied but isn't a known config.
[2020-10-04 02:29:25,869] WARN <ClientConfig:355> [oop-thread-0] The configuration 'ssl.keystore.location' was supplied but isn't a known config.
[2020-10-04 02:29:25,869] WARN <ClientConfig:355> [oop-thread-0] The configuration 'ssl.truststore.password' was supplied but isn't a known config.
[2020-10-04 02:29:25,869] WARN <ClientConfig:355> [oop-thread-0] The configuration 'ssl.endpoint.identification.algorithm' was supplied but isn't a known config.
[2020-10-04 02:29:25,876] INFO <ppInfoParser:117> [oop-thread-0] Kafka version: 2.5.0
[2020-10-04 02:29:25,877] INFO <ppInfoParser:118> [oop-thread-0] Kafka commitId: 66563e712b0b9f84
[2020-10-04 02:29:25,878] INFO <ppInfoParser:119> [oop-thread-0] Kafka startTimeMs: 1601778565869
[2020-10-04 02:29:25,956] INFO <kEventThread:65> [calhost:2181] Starting ZkClient event thread.
[2020-10-04 02:29:25,969] INFO <ZooKeeper :109> [ker-thread-0] Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
[2020-10-04 02:29:25,969] INFO <ZooKeeper :109> [ker-thread-0] Client environment:host.name=sync-cluster-entity-operator-59b97b8c88-zlfrw
[2020-10-04 02:29:25,969] INFO <ZooKeeper :109> [ker-thread-0] Client environment:java.version=11.0.7
[2020-10-04 02:29:25,970] INFO <ZooKeeper :109> [ker-thread-0] Client environment:java.vendor=Oracle Corporation
[2020-10-04 02:29:25,971] INFO <ZooKeeper :109> [ker-thread-0] Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.7.10-4.el7_8.x86_64
[2020-10-04 02:29:25,971] INFO <ZooKeeper :109> [ker-thread-0] Client environment:java.class.path=lib/io.strimzi.topic-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar
[2020-10-04 02:29:25,972] INFO <ZooKeeper :109> [ker-thread-0] Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
[2020-10-04 02:29:25,972] INFO <ZooKeeper :109> [ker-thread-0] Client environment:java.io.tmpdir=/tmp
[2020-10-04 02:29:25,973] INFO <ZooKeeper :109> [ker-thread-0] Client environment:java.compiler=<NA>
[2020-10-04 02:29:25,973] INFO <ZooKeeper :109> [ker-thread-0] Client environment:os.name=Linux
[2020-10-04 02:29:25,973] INFO <ZooKeeper :109> [ker-thread-0] Client environment:os.arch=amd64
[2020-10-04 02:29:25,974] INFO <ZooKeeper :109> [ker-thread-0] Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
[2020-10-04 02:29:25,974] INFO <ZooKeeper :109> [ker-thread-0] Client environment:user.name=strimzi
[2020-10-04 02:29:25,974] INFO <ZooKeeper :109> [ker-thread-0] Client environment:user.home=/home/strimzi
[2020-10-04 02:29:25,974] INFO <ZooKeeper :109> [ker-thread-0] Client environment:user.dir=/opt/strimzi
[2020-10-04 02:29:25,975] INFO <ZooKeeper :109> [ker-thread-0] Client environment:os.memory.free=172MB
[2020-10-04 02:29:25,975] INFO <ZooKeeper :109> [ker-thread-0] Client environment:os.memory.max=3596MB
[2020-10-04 02:29:25,975] INFO <ZooKeeper :109> [ker-thread-0] Client environment:os.memory.total=226MB
[2020-10-04 02:29:25,979] INFO <ZooKeeper :868> [ker-thread-0] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.I0Itec.zkclient.ZkClient@4f20ffb5
[2020-10-04 02:29:25,986] INFO <X509Util :79> [ker-thread-0] Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
[2020-10-04 02:29:25,992] INFO <ntCnxnSocket:237> [ker-thread-0] jute.maxbuffer value is 4194304 Bytes
[2020-10-04 02:29:26,047] INFO <ClientCnxn :1653> [ker-thread-0] zookeeper.request.timeout value is 0. feature enabled=
[2020-10-04 02:29:26,050] INFO <ZkClient :936> [ker-thread-0] Waiting for keeper state SyncConnected
[2020-10-04 02:29:26,070] INFO <ClientCnxn :1112> [alhost:2181)] Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
[2020-10-04 02:29:26,071] INFO <ClientCnxn :1244> [alhost:2181)] Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused
[2020-10-04 02:29:27,174] INFO <ClientCnxn :1112> [alhost:2181)] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[2020-10-04 02:29:27,175] INFO <ClientCnxn :959> [alhost:2181)] Socket connection established, initiating session, client: /127.0.0.1:52458, server: localhost/127.0.0.1:2181
[2020-10-04 02:29:27,229] INFO <ClientCnxn :1394> [alhost:2181)] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x3000012a6080003, negotiated timeout = 20000
[2020-10-04 02:29:27,232] INFO <ZkClient :713> [-EventThread] zookeeper state changed (SyncConnected)
[2020-10-04 02:29:27,261] INFO <opicOperator:1215> [oop-thread-0] Starting initial reconciliation
[2020-10-04 02:29:27,274] INFO <Session :232> [oop-thread-0] Started
[2020-10-04 02:29:27,613] WARN <onUsageUtils:60> [s-ops-tool-0] The client is using resource type 'kafkatopics' with unstable version 'v1beta1'
[2020-10-04 02:29:27,801] WARN <ctionManager:202> [0.96.0.1/...] Exec Failure: HTTP 403, Status: 403 - kafkatopics.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot watch resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,806] ERROR <opicOperator:1351> [oop-thread-1] 0|initial kafka shipments: Error getting KafkaTopic shipments for topic shipments
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/streaming/kafkatopics/shipments. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. kafkatopics.kafka.strimzi.io "shipments" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,820] ERROR <opicOperator:1351> [oop-thread-1] 1|initial kafka sales: Error getting KafkaTopic sales for topic sales
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/streaming/kafkatopics/sales. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. kafkatopics.kafka.strimzi.io "sales" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,821] ERROR <opicOperator:1351> [oop-thread-1] 2|initial kafka users: Error getting KafkaTopic users for topic users
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/streaming/kafkatopics/users. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. kafkatopics.kafka.strimzi.io "users" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-10-04 02:29:27,824] ERROR <Main :63> [oop-thread-1] Error deploying Session
io.fabric8.kubernetes.client.KubernetesClientException: kafkatopics.kafka.strimzi.io is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot watch resource "kafkatopics" in API group "kafka.strimzi.io" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:206) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Suppressed: java.lang.Throwable: waiting here
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:135) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:354) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:759) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:738) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:69) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.strimzi.operator.topic.Session.lambda$start$3(Session.java:197) ~[io.strimzi.topic-operator-0.19.0.jar:0.19.0]
at java.lang.Thread.run(Thread.java:834) [?:?]
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]#
[root@sync-cluster setup-files]# oc logs -f sync-cluster-entity-operator-59b97b8c88-zlfrw -c user-operator
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' false == true ']'
++ echo ''
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
++ java -version
++ sed -E -n 's/.* version "([0-9]*).*$/\1/p'
+ JAVA_MAJOR_VERSION=11
+ '[' 11 -ge 9 ']'
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.user-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.10.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar io.strimzi.operator.user.Main
[2020-10-04 02:27:58,314] INFO <Main :54> [main ] UserOperator 0.19.0 is starting
[2020-10-04 02:27:58,957] INFO <Util :268> [main ] Using config:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
STRIMZI_CA_RENEWAL: 30
SYNC_CLUSTER_KAFKA_2_SERVICE_PORT: 9094
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP: tcp://10.96.164.152:9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP: tcp://10.96.142.104:9094
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT: tcp://10.96.164.152:9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
KAFKA_ACL_VIEWER_SERVICE_HOST: 10.96.208.142
STRIMZI_EO_KEY_SECRET_NAME: sync-cluster-entity-operator-certs
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT: tcp://10.96.98.125:9091
STRIMZI_HOME: /opt/strimzi
PWD: /opt/strimzi
SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT_REST_API: 8080
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP: tcp://10.96.38.96:8080
JAVA_MAIN: io.strimzi.operator.user.Main
AKHQ_PORT_80_TCP_PORT: 80
STRIMZI_NAMESPACE: streaming
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_HOST: 10.96.164.152
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PROTO: tcp
STRIMZI_CA_VALIDITY: 365
KAFKA_ACL_VIEWER_PORT: tcp://10.96.208.142:80
KAFKA_ACL_VIEWER_PORT_80_TCP: tcp://10.96.208.142:80
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
SYNC_CLUSTER_KAFKA_1_SERVICE_PORT: 9094
KUBERNETES_SERVICE_PORT_HTTPS: 443
AKHQ_PORT: tcp://10.96.4.192:80
SHLVL: 0
AKHQ_PORT_80_TCP: tcp://10.96.4.192:80
AKHQ_SERVICE_PORT: 80
KAFKA_ACL_VIEWER_PORT_80_TCP_ADDR: 10.96.208.142
TINI_SHA256: 12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855
AKHQ_SERVICE_PORT_HTTP: 80
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP: tcp://10.96.151.227:9094
KUBERNETES_PORT: tcp://10.96.0.1:443
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 10.96.98.125
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
JAVA_CLASSPATH: lib/io.strimzi.user-operator-0.19.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.api-0.19.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.strimzi.certificate-manager-0.19.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.10.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.19.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/io.strimzi.crd-annotations-0.19.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/org.apache.kafka.kafka-clients-2.5.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.7.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.7.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar
SYNC_CLUSTER_KAFKA_2_SERVICE_HOST: 10.96.151.227
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://10.96.16.2:2181
SYNC_CLUSTER_KAFKA_0_SERVICE_HOST: 10.96.142.104
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.96.98.125:9092
KAFKA_ACL_VIEWER_PORT_80_TCP_PROTO: tcp
SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_HOST: 10.96.16.2
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_ADDR: 10.96.151.227
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PORT: 8080
KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_ADDR: 10.96.100.188
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_ADDR: 10.96.142.104
STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 6000
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP: tcp://10.96.100.188:9094
KUBERNETES_PORT_443_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT: 2181
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_ADDR: 10.96.164.152
SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_PORT: 8080
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_PROTO: tcp
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
STRIMZI_LABELS: strimzi.io/cluster=sync-cluster
JAVA_OPTS: -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny
SYNC_CLUSTER_KAFKA_2_PORT: tcp://10.96.151.227:9094
SYNC_CLUSTER_KAFKA_1_PORT: tcp://10.96.100.188:9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.96.98.125
STRIMZI_GC_LOG_ENABLED: false
SYNC_CLUSTER_KAFKA_1_SERVICE_HOST: 10.96.100.188
KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT: tcp://10.96.16.2:2181
STRIMZI_VERSION: 0.19.0
STRIMZI_KAFKA_BOOTSTRAP_SERVERS: sync-cluster-kafka-bootstrap:9091
SYNC_CLUSTER_KAFKA_0_PORT: tcp://10.96.142.104:9094
AKHQ_PORT_80_TCP_PROTO: tcp
AKHQ_SERVICE_HOST: 10.96.4.192
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT: 9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTSTLS: 9093
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PROTO: tcp
STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
AKHQ_PORT_80_TCP_ADDR: 10.96.4.192
SYNC_CLUSTER_KAFKA_0_SERVICE_PORT: 9094
SYNC_BRIDGE_BRIDGE_SERVICE_PORT: tcp://10.96.38.96:8080
SYNC_CLUSTER_KAFKA_0_SERVICE_PORT_TCP_EXTERNAL: 9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.96.98.125
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.96.0.1
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://10.96.98.125:9093
SYNC_CLUSTER_KAFKA_1_SERVICE_PORT_TCP_EXTERNAL: 9094
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PORT: 9094
STRIMZI_CLUSTER_CA_CERT_SECRET_NAME: sync-cluster-cluster-ca-cert
SYNC_BRIDGE_BRIDGE_SERVICE_SERVICE_HOST: 10.96.38.96
TINI_VERSION: v0.18.0
KAFKA_ACL_VIEWER_SERVICE_PORT_80_TCP: 80
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.96.98.125
KAFKA_ACL_VIEWER_SERVICE_PORT: 80
STRIMZI_CA_CERT_NAME: sync-cluster-clients-ca-cert
STRIMZI_CA_NAMESPACE: streaming
SYNC_CLUSTER_ZOOKEEPER_CLIENT_SERVICE_PORT_TCP_CLIENTS: 2181
SYNC_CLUSTER_KAFKA_2_PORT_9094_TCP_PORT: 9094
KUBERNETES_SERVICE_PORT: 443
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.96.98.125:9091
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PORT: 9094
SYNC_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
SYNC_CLUSTER_KAFKA_1_PORT_9094_TCP_PORT: 9094
SYNC_CLUSTER_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 10.96.16.2
HOSTNAME: sync-cluster-entity-operator-59b97b8c88-zlfrw
STRIMZI_CA_KEY_NAME: sync-cluster-clients-ca
KUBERNETES_PORT_443_TCP_PORT: 443
KAFKA_ACL_VIEWER_PORT_80_TCP_PORT: 80
SYNC_BRIDGE_BRIDGE_SERVICE_PORT_8080_TCP_ADDR: 10.96.38.96
SYNC_CLUSTER_KAFKA_2_SERVICE_PORT_TCP_EXTERNAL: 9094
HOME: /home/strimzi
SYNC_CLUSTER_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT_TCP_EXTERNAL: 9094
SYNC_CLUSTER_KAFKA_0_PORT_9094_TCP_PROTO: tcp
MALLOC_ARENA_MAX: 2
[2020-10-04 02:27:59,624] ERROR <Main :75> [oop-thread-0] Unable to start operator
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/streaming/secrets/sync-cluster-cluster-ca-cert. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. secrets "sync-cluster-cluster-ca-cert" is forbidden: User "system:serviceaccount:streaming:sync-cluster-entity-operator" cannot get resource "secrets" in API group "" in the namespace "streaming": RBAC: clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:337) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:318) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:833) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:226) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:170) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.get(AbstractResourceOperator.java:210) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$getAsync$1(AbstractResourceOperator.java:223) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
Issue Analytics
- State:
- Created 3 years ago
- Comments:9 (1 by maintainers)
Top Results From Across the Web
Posts | Veerendra's Blog
Today's guide is on how to recover from a disaster for Strimzi Kafka with Velero. First of all, what is Strmzi Kafka? https://strimzi.io...
Read more >Velero Docs - Troubleshooting
Velero reports custom resource not found errors when starting up. velero backup logs returns a SignatureDoesNotMatch error.
Read more >Configuring Strimzi (0.32.0)
Recovery of data in the event of a system failure. Aggregation of data for analysis. Restriction of data access to a specific cluster....
Read more >Velero Deployment with Kustomize (Azure) - Medium
Hello guys, today I came up with an interesting write up, that is how to setup backup and restore with Velero on Kubernetes....
Read more >Newest 'velero' Questions - Stack Overflow
velero with velero-plugin-for-aws backup jobs failed. k3s cluster. I have used velero helm ... I'm trying to setup backup/restore for Strimzi Kafka.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@cloudcafetech Does backup also follow sequence? Can paste velero restore commands to restore in mentioned sequence?
Sequence as follows …
PVC Cluster Operator Topics Kafka Cluster Users