Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
[Question] Standalone topic operator with AWS MSK (TLS error)
See original GitHub issueI stood up a standalone topic operator following the docs (latest 0.17.0) and I’m trying to have it communicate with an MSK cluster.
The MSK cluster is configured to allow only TLS connections to the brokers.
If I just set STRIMZI_TLS_ENABLED to true, the startup scripts fail because it expects the certificates in the tls-sidecar path, but those directories do not exist (there is no side car running, nor I could find instructions on how to do so, but I assume that the sidecar gets deployed when using the topic operator within the cluster operator).
Looking at the startup scripts it seems that the logic to create a keystore is only executed if the truststore and keystore variables are not used.
The upstream AWS MSK docs suggest to use the default truststore since the certificates used in the MSK clusters have public certs.
So I tried setting STRIMZI_TRUSTSTORE_LOCATION to /usr/lib/jvm/jre/lib/security/cacerts which, after a series of symlink, resolve to the correct file (I tried listing the content with keytool with an empty password and the list of CAs is pretty big).
this is the startup logs (and failure):
➭ kubectl logs strimzi-topic-operator-msk-0-575b9d9df7-jbpkh
+ shift
+ . /opt/strimzi/bin/dynamic_resources.sh
++ get_heap_size
+++ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
++ CONTAINER_MEMORY_IN_BYTES=524288000
++ DEFAULT_MEMORY_CEILING=32505856
++ '[' 524288000 -lt 32505856 ']'
+ MAX_HEAP=
+ '[' -n '' ']'
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
+ exec /usr/bin/tini -w -e 143 -- java -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -classpath lib/io.strimzi.topic-operator-0.17.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/com.github.luben.zstd-jni-1.4.3-1.jar:lib/io.netty.netty-handler-4.1.45.Final.jar:lib/com.101tec.zkclient-0.11.jar:lib/io.netty.netty-codec-http-4.1.45.Final.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.netty.netty-buffer-4.1.45.Final.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-common-4.1.45.Final.jar:lib/org.apache.logging.log4j.log4j-api-2.13.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/io.netty.netty-codec-4.1.45.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.strimzi.certificate-manager-0.17.0.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.vertx.vertx-core-3.8.5.jar:lib/io.netty.netty-codec-dns-4.1.45.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/io.netty.netty-codec-socks-4.1.45.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.6.jar:lib/io.netty.netty-resolver-4.1.45.Final.jar:lib/io.netty.netty-handler-proxy-4.1.45.Final.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.strimzi.api-0.17.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.45.Final.jar:lib/org.apache.zookeeper.zookeeper-3.5.6.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.vertx.vertx-micrometer-metrics-3.8.5.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/io.netty.netty-transport-4.1.45.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.45.Final.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.lz4.lz4-java-1.6.0.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-codec-http2-4.1.45.Final.jar:lib/io.strimzi.operator-common-0.17.0.jar:lib/org.apache.logging.log4j.log4j-core-2.13.0.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/io.netty.netty-resolver-dns-4.1.45.Final.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.crd-annotations-0.17.0.jar:lib/org.apache.kafka.kafka-clients-2.4.0.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar io.strimzi.operator.topic.Main
2020-03-29 04:48:52 INFO Main:30 - TopicOperator 0.17.0 is starting
2020-03-29 04:48:52 DEBUG Config:465 - Trying to configure client from Kubernetes config...
2020-03-29 04:48:52 DEBUG Config:491 - Did not find Kubernetes config at: [/home/strimzi/.kube/config]. Ignoring.
2020-03-29 04:48:52 DEBUG Config:395 - Trying to configure client from service account...
2020-03-29 04:48:52 DEBUG Config:400 - Found service account host and port: 100.64.0.1:443
2020-03-29 04:48:52 DEBUG Config:406 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt].
2020-03-29 04:48:52 DEBUG Config:414 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2020-03-29 04:48:52 DEBUG Config:605 - Trying to configure client namespace from Kubernetes service account namespace path...
2020-03-29 04:48:52 DEBUG Config:610 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2020-03-29 04:48:53 DEBUG InternalLoggerFactory:45 - Using SLF4J as the default logging framework
2020-03-29 04:48:53 DEBUG ResourceLeakDetector:130 - -Dio.netty.leakDetection.level: simple
2020-03-29 04:48:53 DEBUG ResourceLeakDetector:131 - -Dio.netty.leakDetection.targetRecords: 4
2020-03-29 04:48:53 DEBUG InternalThreadLocalMap:54 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2020-03-29 04:48:53 DEBUG InternalThreadLocalMap:57 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2020-03-29 04:48:53 DEBUG MultithreadEventLoopGroup:44 - -Dio.netty.eventLoopThreads: 2
2020-03-29 04:48:53 DEBUG NioEventLoop:106 - -Dio.netty.noKeySetOptimization: false
2020-03-29 04:48:53 DEBUG NioEventLoop:107 - -Dio.netty.selectorAutoRebuildThreshold: 512
2020-03-29 04:48:53 DEBUG PlatformDependent0:396 - -Dio.netty.noUnsafe: false
2020-03-29 04:48:53 DEBUG PlatformDependent0:852 - Java version: 8
2020-03-29 04:48:53 DEBUG PlatformDependent0:121 - sun.misc.Unsafe.theUnsafe: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:145 - sun.misc.Unsafe.copyMemory: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:183 - java.nio.Buffer.address: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:244 - direct buffer constructor: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:314 - java.nio.Bits.unaligned: available, true
2020-03-29 04:48:53 DEBUG PlatformDependent0:379 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
2020-03-29 04:48:53 DEBUG PlatformDependent0:386 - java.nio.DirectByteBuffer.<init>(long, int): available
2020-03-29 04:48:53 DEBUG PlatformDependent:1030 - sun.misc.Unsafe: available
2020-03-29 04:48:53 DEBUG PlatformDependent:1149 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2020-03-29 04:48:53 DEBUG PlatformDependent:1228 - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2020-03-29 04:48:53 DEBUG PlatformDependent:174 - -Dio.netty.maxDirectMemory: 127729664 bytes
2020-03-29 04:48:53 DEBUG PlatformDependent:181 - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2020-03-29 04:48:53 DEBUG CleanerJava6:92 - java.nio.ByteBuffer.cleaner(): available
2020-03-29 04:48:53 DEBUG PlatformDependent:201 - -Dio.netty.noPreferDirect: false
2020-03-29 04:48:53 DEBUG PlatformDependent:891 - org.jctools-core.MpscChunkedArrayQueue: available
2020-03-29 04:48:53 DEBUG DefaultDnsServerAddressStreamProvider:82 - Default DNS servers: [/100.64.0.10:53] (sun.net.dns.ResolverConfiguration)
2020-03-29 04:48:53 DEBUG NetUtil:139 - -Djava.net.preferIPv4Stack: false
2020-03-29 04:48:53 DEBUG NetUtil:140 - -Djava.net.preferIPv6Addresses: false
2020-03-29 04:48:53 DEBUG NetUtil:224 - Loopback interface: lo (lo, 127.0.0.1)
2020-03-29 04:48:53 DEBUG NetUtil:271 - /proc/sys/net/core/somaxconn: 128
2020-03-29 04:48:53 INFO Session:70 - Using config:
STRIMZI_TRUSTSTORE_LOCATION: /usr/lib/jvm/jre/lib/security/cacerts
STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=cxp-msk-0
STRIMZI_KAFKA_BOOTSTRAP_SERVERS: b-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094,b-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094,b-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094
STRIMZI_NAMESPACE: cxp
STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 20000
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 900000
STRIMZI_ZOOKEEPER_CONNECT: z-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:2181,z-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:2181,z-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:2181
STRIMZI_TLS_ENABLED: true
STRIMZI_KEYSTORE_PASSWORD:
STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
STRIMZI_KEYSTORE_LOCATION:
TC_ZK_CONNECTION_TIMEOUT_MS: 20000
STRIMZI_TRUSTSTORE_PASSWORD:
STRIMZI_REASSIGN_THROTTLE: 9223372036854775807
2020-03-29 04:48:53 DEBUG InternalLoggerFactory:61 - Using SLF4J as the default logging framework
2020-03-29 04:48:53 INFO Session:141 - Starting
2020-03-29 04:48:53 INFO AdminClientConfig:347 - AdminClientConfig values:
bootstrap.servers = [b-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094, b-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094, b-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 120000
retries = 5
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = HTTPS
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location =
ssl.keystore.password = [hidden]
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = /usr/lib/jvm/jre/lib/security/cacerts
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
2020-03-29 04:48:54 DEBUG AdminMetadataManager:248 - [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [b-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094 (id: -3 rack: null), b-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094 (id: -2 rack: null), b-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094 (id: -1 rack: null)], partitions = [], controller = null).
2020-03-29 04:48:54 ERROR Main:55 - Error deploying Session
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.admin.Admin.create(Admin.java:59) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at io.strimzi.operator.topic.Session.start(Session.java:157) ~[io.strimzi.topic-operator-0.17.0.jar:0.17.0]
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$9(DeploymentManager.java:556) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:369) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore of type JKS
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
... 13 more
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore of type JKS
at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
... 13 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore of type JKS
at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
... 13 more
Caused by: java.io.IOException: Is a directory
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_242]
at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46) ~[?:1.8.0_242]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_242]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_242]
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159) ~[?:1.8.0_242]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:1.8.0_242]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:1.8.0_242]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:1.8.0_242]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_242]
at java.io.BufferedInputStream.read(BufferedInputStream.java:265) ~[?:1.8.0_242]
at java.security.DigestInputStream.read(DigestInputStream.java:124) ~[?:1.8.0_242]
at java.io.DataInputStream.readInt(DataInputStream.java:387) ~[?:1.8.0_242]
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:658) ~[?:1.8.0_242]
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) ~[?:1.8.0_242]
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[?:1.8.0_242]
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) ~[?:1.8.0_242]
at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_242]
at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
... 13 more
If I disable TLS, I see zookeeper being able to list the topics but the K8sTopicWatcher does not go past this:
2020-03-29 04:33:12 DEBUG K8sTopicWatcher:40 - Ignoring initial event for KafkaTopic cxp-strimzi-test during initial reconcile
Any help would be greatly appreciated.
Issue Analytics
- State:
- Created 3 years ago
- Comments:11 (5 by maintainers)
Top Results From Across the Web
Troubleshoot issues when connecting to your Amazon MSK ...
You might get this error when the producer or consumer tries to connect to a TLS-encrypted cluster over TLS port 9094 without passing...
Read more >Troubleshooting your Amazon MSK cluster
The following information can help you troubleshoot problems that you might have with your Amazon MSK cluster. You can also post your issue...
Read more >Mutual TLS authentication - Amazon Managed Streaming for ...
Amazon MSK doesn't support certificate revocation lists (CRLs). To control access to your cluster topics or block compromised certificates, use Apache Kafka ...
Read more >Deploying and Upgrading (In Development) - Strimzi
On deployment, the Cluster Operator automatically sets up TLS ... The Topic Operator and User Operator standalone if you did not deploy them...
Read more >Amazon Managed Streaming for Apache Kafka (MSK)
If you're using the PLAINTEXT Kafka host information, this will be on port 9092, and TLS brokers will be on port 9094. When...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Just for whoever stumbles upon this, I was able to make this work without modifications (ie: with MSK using TLS but not client cert auth). I patched the deployment (we use kustomize) to add an init container that creates the trust/key store (we don’t have any private key so we point both variables to the same store, but it could be separate I guess, I haven’t tested that). Both the init container and the topic operator container use the same image and they both mount an emptyDir volume to have access to the same files.
all the init container does is:
Then in the main container I pass the vars:
Given they are separate containers, I couldn’t create the stores with a random password without sharing it somehow. This could be improved if somehow the startup scripts would include some logic to accommodate this scenario (so that random passwords could be used), but for as long as i’m concerned this issue can be closed.
@dsouzat Could you share your solution please?