[cetic/nifi] cert-manager not correctly generating the ca.cert
See original GitHub issueDescribe the bug When trying the new cert-manager I get the following error.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
Version of Helm, Kubernetes and the Nifi chart:
helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.18.2"}
kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.19) exceeds the supported minor version skew of +/-1
NiFi chart version: 1.1.1
What happened:
I tested the new cert-manager configuration but the nifi pods are failing to start as the container cert-manager is failing constantly complaining about an input not an x.509 certificate cert-manager
error.
What you expected to happen: The cert-manager to provide a valid ca certificate and the cluster to run normally.
How to reproduce it (as minimally and precisely as possible):
I basically copied the same values as the ones reported in https://github.com/cetic/helm-nifi/issues/224#issuecomment-1023663727.
I also modified the statefulset.yaml file as shown here in line 199.
-- --value "Initial User Identity {{ . }}" \
++ --value "Initial User Identity {{ add 2 . }}" \
Anything else we need to know:
If I ssh into the server pod and check the ca.crt then just ??e
is shown.
nifi@nifi-cluster-0:/opt/nifi/nifi-current/tls/cert-manager$ cat ca.crt
??e
Here are some information that help troubleshooting:
- if relevant, provide your
values.yaml
or the changes made to the default one (after removing sensitive information) - the output of the folowing commands:
Check if a pod is in error:
kubectl get pod
NAME READY STATUS RESTARTS AGE
nifi-cluster-0 3/5 CrashLoopBackOff 9 8m43s
nifi-cluster-1 3/5 CrashLoopBackOff 9 11m
nifi-cluster-2 3/5 CrashLoopBackOff 10 11m
nifi-cluster-nifi-registry-0 1/1 Running 0 7d
nifi-cluster-zookeeper-0 1/1 Running 0 11m
nifi-cluster-zookeeper-1 1/1 Running 0 11m
nifi-cluster-zookeeper-2 1/1 Running 0 11m
Inspect the pod, check the “Events” section at the end for anything suspicious.
kubectl describe pod myrelease-nifi-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned nifi-cluster/nifi-cluster-1 to worker3.x.k8s-test.x.io
Normal Pulled 12m kubelet Container image "busybox:1.32.0" already present on machine
Normal Created 12m kubelet Created container zookeeper
Normal Started 12m kubelet Started container zookeeper
Normal Pulling 12m kubelet Pulling image "apache/nifi:1.16.3"
Normal Pulled 11m kubelet Successfully pulled image "apache/nifi:1.16.3" in 39.432102597s
Normal Pulled 11m kubelet Container image "busybox:1.32.0" already present on machine
Normal Created 11m kubelet Created container user-log
Normal Pulled 11m kubelet Container image "busybox:1.32.0" already present on machine
Normal Created 11m kubelet Created container app-log
Normal Started 11m kubelet Started container app-log
Normal Started 11m kubelet Started container user-log
Normal Created 11m kubelet Created container bootstrap-log
Normal Started 11m kubelet Started container bootstrap-log
Normal Pulled 11m kubelet Container image "busybox:1.32.0" already present on machine
Normal Started 11m (x2 over 11m) kubelet Started container server
Normal Created 11m (x2 over 11m) kubelet Created container server
Normal Pulled 11m kubelet Container image "apache/nifi:1.16.3" already present on machine
Normal Pulled 11m (x2 over 11m) kubelet Container image "apache/nifi:1.16.3" already present on machine
Normal Created 11m (x2 over 11m) kubelet Created container cert-manager
Normal Started 11m (x2 over 11m) kubelet Started container cert-manager
Warning BackOff 2m34s (x41 over 11m) kubelet Back-off restarting failed container
Get logs on a failed container inside the pod (here the server
one):
kubectl logs nifi-cluster-1 server
Java home: /usr/local/openjdk-8
NiFi home: /opt/nifi/nifi-current
Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf
Login Identity Providers Processed [/opt/nifi/nifi-current/./conf/login-identity-providers.xml]
updating nifi.remote.input.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.cluster.node.address in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.zookeeper.connect.string in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.http.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.proxy.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.keystore in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.keystoreType in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.keystorePasswd in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.truststore in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.truststoreType in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.truststorePasswd in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.https.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.cluster.node.address in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.https.network.interface.default in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.https.network.interface.lo in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.http.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.http.port in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.autoreload.enabled in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.autoreload.interval in /opt/nifi/nifi-current/conf/nifi.properties
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it. (infinite loop)
kubectl logs nifi-cluster-1 cert-manager
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 257 100 257 0 0 2519 0 --:--:-- --:--:-- --:--:-- 23363
keytool error: java.lang.Exception: Input not an X.509 certificate
Issue Analytics
- State:
- Created a year ago
- Comments:6 (3 by maintainers)
@lfreinag looks like your Common Name
nifi-cluster-0.nifi-cluster-headless.nifi-cluster.svc.cluster.local
is 67 characters long, which is beyond the limit of 64 characters. That’s at least one of the reasons why cert-manager isn’t issuing a certificate, and therefore isn’t populating the secret. You may need to choose shorter Helm deployment and/or namespace names to get those Common Names down to 64 characters.This chart targets the cert-manager.io/v1 API, which first became available in cert-manager version 1.0 (see https://cert-manager.io/docs/release-notes/release-notes-1.0). Also see https://github.com/cetic/helm-nifi/blob/master/tests/05-install-cert-manager.bash for how the chart regression tests install the latest version of cert-manager.
So I now installed from scratch v1.0.0 for cert-manager and it is working like a charm! 🎉 Thanks for your support here. Great to see that multi cluster node setup is working too.
I am closing the issue. Have a nice weekend!