[cetic/nifi] cert-manager not correctly generating the ca.cert

Describe the bug When trying the new cert-manager I get the following error.

/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.

Version of Helm, Kubernetes and the Nifi chart:

helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.18.2"}

kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.19) exceeds the supported minor version skew of +/-1

NiFi chart version: 1.1.1

What happened: I tested the new cert-manager configuration but the nifi pods are failing to start as the container cert-manager is failing constantly complaining about an input not an x.509 certificate cert-manager error.

What you expected to happen: The cert-manager to provide a valid ca certificate and the cluster to run normally.

How to reproduce it (as minimally and precisely as possible):

I basically copied the same values as the ones reported in https://github.com/cetic/helm-nifi/issues/224#issuecomment-1023663727.

I also modified the statefulset.yaml file as shown here in line 199.

--            --value "Initial User Identity {{ . }}" \
++            --value "Initial User Identity {{ add 2 . }}" \

Anything else we need to know:

If I ssh into the server pod and check the ca.crt then just ??e is shown.

nifi@nifi-cluster-0:/opt/nifi/nifi-current/tls/cert-manager$ cat ca.crt 
??e

Here are some information that help troubleshooting:

• if relevant, provide your values.yaml or the changes made to the default one (after removing sensitive information)
• the output of the folowing commands:

Check if a pod is in error:

kubectl get pod
NAME                           READY   STATUS             RESTARTS   AGE
nifi-cluster-0                 3/5     CrashLoopBackOff   9          8m43s
nifi-cluster-1                 3/5     CrashLoopBackOff   9          11m
nifi-cluster-2                 3/5     CrashLoopBackOff   10         11m
nifi-cluster-nifi-registry-0   1/1     Running            0          7d
nifi-cluster-zookeeper-0       1/1     Running            0          11m
nifi-cluster-zookeeper-1       1/1     Running            0          11m
nifi-cluster-zookeeper-2       1/1     Running            0          11m

Inspect the pod, check the “Events” section at the end for anything suspicious.

kubectl describe pod myrelease-nifi-0
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned nifi-cluster/nifi-cluster-1 to worker3.x.k8s-test.x.io
  Normal   Pulled     12m                   kubelet            Container image "busybox:1.32.0" already present on machine
  Normal   Created    12m                   kubelet            Created container zookeeper
  Normal   Started    12m                   kubelet            Started container zookeeper
  Normal   Pulling    12m                   kubelet            Pulling image "apache/nifi:1.16.3"
  Normal   Pulled     11m                   kubelet            Successfully pulled image "apache/nifi:1.16.3" in 39.432102597s
  Normal   Pulled     11m                   kubelet            Container image "busybox:1.32.0" already present on machine
  Normal   Created    11m                   kubelet            Created container user-log
  Normal   Pulled     11m                   kubelet            Container image "busybox:1.32.0" already present on machine
  Normal   Created    11m                   kubelet            Created container app-log
  Normal   Started    11m                   kubelet            Started container app-log
  Normal   Started    11m                   kubelet            Started container user-log
  Normal   Created    11m                   kubelet            Created container bootstrap-log
  Normal   Started    11m                   kubelet            Started container bootstrap-log
  Normal   Pulled     11m                   kubelet            Container image "busybox:1.32.0" already present on machine
  Normal   Started    11m (x2 over 11m)     kubelet            Started container server
  Normal   Created    11m (x2 over 11m)     kubelet            Created container server
  Normal   Pulled     11m                   kubelet            Container image "apache/nifi:1.16.3" already present on machine
  Normal   Pulled     11m (x2 over 11m)     kubelet            Container image "apache/nifi:1.16.3" already present on machine
  Normal   Created    11m (x2 over 11m)     kubelet            Created container cert-manager
  Normal   Started    11m (x2 over 11m)     kubelet            Started container cert-manager
  Warning  BackOff    2m34s (x41 over 11m)  kubelet            Back-off restarting failed container

Get logs on a failed container inside the pod (here the server one):

kubectl logs nifi-cluster-1 server

Java home: /usr/local/openjdk-8
NiFi home: /opt/nifi/nifi-current

Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf

Login Identity Providers Processed [/opt/nifi/nifi-current/./conf/login-identity-providers.xml]

updating nifi.remote.input.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.cluster.node.address in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.zookeeper.connect.string in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.http.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.proxy.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.keystore in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.keystoreType in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.keystorePasswd in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.truststore in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.truststoreType in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.truststorePasswd in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.https.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.cluster.node.address in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.https.network.interface.default in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.https.network.interface.lo in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.http.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.http.port in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.autoreload.enabled in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.security.autoreload.interval in /opt/nifi/nifi-current/conf/nifi.properties
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.
/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it. (infinite loop)

kubectl logs nifi-cluster-1 cert-manager
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   257  100   257    0     0   2519      0 --:--:-- --:--:-- --:--:-- 23363
keytool error: java.lang.Exception: Input not an X.509 certificate

[wknickless]

@lfreinag looks like your Common Name nifi-cluster-0.nifi-cluster-headless.nifi-cluster.svc.cluster.local is 67 characters long, which is beyond the limit of 64 characters. That’s at least one of the reasons why cert-manager isn’t issuing a certificate, and therefore isn’t populating the secret. You may need to choose shorter Helm deployment and/or namespace names to get those Common Names down to 64 characters.

This chart targets the cert-manager.io/v1 API, which first became available in cert-manager version 1.0 (see https://cert-manager.io/docs/release-notes/release-notes-1.0). Also see https://github.com/cetic/helm-nifi/blob/master/tests/05-install-cert-manager.bash for how the chart regression tests install the latest version of cert-manager.

[lfreinag]

So I now installed from scratch v1.0.0 for cert-manager and it is working like a charm! 🎉 Thanks for your support here. Great to see that multi cluster node setup is working too.

I am closing the issue. Have a nice weekend!