question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Deploying Standalone User Operator CA Secret Format

See original GitHub issue

Suggestion / Problem I was trying to deploy the Standalone User Operator against AWS MSK and using TLS as Auth. For this I created an AWS Private certificate authority, and then a Subordinate CA that could issue certificates. I added them as secrets using

kubectl create --save-config --dry-run=client secret generic strimzi-subordinate-cert --from-file=keys/subordinate.crt -o yaml | kubectl apply -f -
kubectl create --save-config --dry-run=client secret generic strimzi-subordinate-key --from-file=keys/subordinate.key -o yaml | kubectl apply -f -

and tried my operator but it was failing with a NPE pointing at https://github.com/strimzi/strimzi-kafka-operator/blob/main/operator-common/src/main/java/io/strimzi/operator/cluster/model/Ca.java#L731.

2021-06-30 21:53:09 DEBUG Ca:282 - Reconciliation #3(timer) KafkaUser(default/kafka.user.kafka-simple-client): Generating certificate /CN=kafka.user.kafka-simple-client with SAN null, signed by CA clients-ca
2021-06-30 21:53:09 DEBUG OpenSslCertManager:617 - Running command [openssl, req, -new, -batch, -nodes, -keyout, /tmp/tls5600546536714548180key, -out, /tmp/tls7779136491457340388csr, -subj, /CN=kafka.user.kafka-simple-client]
2021-06-30 21:53:09 DEBUG OpenSslCertManager:640 - Got result 0
2021-06-30 21:53:09 WARN  KafkaUserOperator:149 - Reconciliation #3(timer) KafkaUser(default/kafka.user.kafka-simple-client): java.lang.NullPointerException
2021-06-30 21:53:09 ERROR AbstractOperator:274 - Reconciliation #3(timer) KafkaUser(default/kafka.user.kafka-simple-client): createOrUpdate failed
java.lang.NullPointerException: null
	at java.util.Base64$Decoder.decode(Base64.java:561) ~[?:?]
	at io.strimzi.operator.cluster.model.Ca.currentCaKey(Ca.java:732) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.cluster.model.Ca.generateSignedCert(Ca.java:285) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.cluster.model.Ca.generateSignedCert(Ca.java:332) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.cluster.model.Ca.generateSignedCert(Ca.java:307) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.user.model.KafkaUserModel.maybeGenerateCertificates(KafkaUserModel.java:245) ~[io.strimzi.user-operator-0.24.0.jar:0.24.0]
	at io.strimzi.operator.user.model.KafkaUserModel.fromCrd(KafkaUserModel.java:125) ~[io.strimzi.user-operator-0.24.0.jar:0.24.0]
	at io.strimzi.operator.user.operator.KafkaUserOperator.createOrUpdate(KafkaUserOperator.java:147) ~[io.strimzi.user-operator-0.24.0.jar:0.24.0]
	at io.strimzi.operator.user.operator.KafkaUserOperator.createOrUpdate(KafkaUserOperator.java:45) ~[io.strimzi.user-operator-0.24.0.jar:0.24.0]
	at io.strimzi.operator.common.AbstractOperator.lambda$reconcile$7(AbstractOperator.java:255) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.common.AbstractOperator.lambda$withLock$15(AbstractOperator.java:401) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:124) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:62) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureImpl.addListener(FutureImpl.java:164) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.PromiseImpl.addListener(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureImpl.onComplete(FutureImpl.java:132) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.PromiseImpl.onComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.shareddata.impl.SharedDataImpl.getLockWithTimeout(SharedDataImpl.java:100) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.strimzi.operator.common.AbstractOperator.withLock(AbstractOperator.java:391) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.common.AbstractOperator.reconcile(AbstractOperator.java:197) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.common.Operator.reconcileThese(Operator.java:71) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.strimzi.operator.common.Operator.lambda$reconcileAll$0(Operator.java:56) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
	at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:124) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:62) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:179) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.Mapping.onSuccess(Mapping.java:40) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:62) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:179) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.CompositeFutureImpl.complete(CompositeFutureImpl.java:172) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.CompositeFutureImpl.lambda$join$3(CompositeFutureImpl.java:109) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:124) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:62) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:179) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.PromiseImpl.onSuccess(PromiseImpl.java:49) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

I could not figure out what was going on, and started to look at base64 re-encoding the certificate, so I searched the docs to find that and founhttps://strimzi.io/docs/operators/in-development/using.html#proc-accessing-kafka-using-nodeports-str which casually mentions kubectl get secret KAFKA-CLUSTER-NAME-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt. With that I relized the key the secret had to have, and then I found https://github.com/strimzi/strimzi-kafka-operator/blob/main/operator-common/src/main/java/io/strimzi/operator/cluster/model/Ca.java#L85-L86 which confirmed this.

I think it would be really helpful to add a simple hint to this in the linked document.

Additionally, it would be great to catch the error and show something more indicative of the problem.

Documentation Link https://strimzi.io/docs/operators/in-development/deploying.html#deploying-the-user-operator-standalone-str

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
scholzjcommented, Jul 1, 2021

I opened #5237 to fix the NPE.

0reactions
scholzjcommented, Jul 1, 2021

Ok great, thanks for the clarification.

On a closing comment, even if not missing from the docs, it might a good idea to catch this error and print a message about “unable to find key CA.crt” to the logs.

I agree that it should do something better than NPE. I will have a look at it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Configuring Strimzi (In Development)
Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations.
Read more >
Chapter 6. Deploying AMQ Streams using installation artifacts
Deploying a standalone Topic Operator and User Operator ... The OpenShift Secret that contains the public key ( ca.crt ) value of the...
Read more >
Providing sensitive data to pods - OpenShift Documentation
The Secret object type provides a mechanism to hold sensitive information such as passwords, OpenShift Container Platform client configuration files, ...
Read more >
Operator pattern - Kubernetes
The most common way to deploy an operator is to add the Custom Resource Definition and its associated Controller to your cluster. The...
Read more >
Deploy an Ops Manager Resource — MongoDB Kubernetes ...
Considerations; Procedure. You can deploy Ops Manager in a container with the Kubernetes Operator. ... The SAN for each pod must use the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found