Unable to start Che by chectl with default TLS - keycloak certificate could not be validated

Describe the bug

After default TLS was added to chectl, I’m unable to start Che. All pods start, but Che pod is stuck in infinite restart loop. Logs reveal

1. Error injecting constructor, java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.10.48.188.27.nip.io/auth/realms/che/.well-known/openid-configuration Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.10.48.188.27.nip.io/auth/realms/che/.well-known/openid-configuration Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

1. Start Che using chectl server:start --multiuser --platform=minikube
2. Wait till startup fails - che pod never gets ready

Eclipse Che pod bootstrap ✔ scheduling…done. ✔ downloading images…done. ✖ starting → ERR_TIMEOUT: Timeout set to pod ready timeout 130000 Retrieving Eclipse Che server URL Eclipse Che status check › Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000 › Installation failed, check logs in ‘/tmp/chectl-logs/1584613990270’

1. Che pods never gets ready - stuck in infinite restarts loop

[root@czprapd-chenext ~]# kubectl get all --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE che pod/che-5968b96bfd-smnnj 0/1 Running 1 7m11s che pod/che-operator-844f4bd4f9-jvvsf 1/1 Running 0 9m3s che pod/devfile-registry-79945cb69f-2bjpk 1/1 Running 0 7m34s che pod/keycloak-56c8774bbf-qp59p 1/1 Running 0 8m33s che pod/plugin-registry-658bb57ff5-fqj5g 1/1 Running 0 7m21s che pod/postgres-984c4cd5c-t2nr8 1/1 Running 0 8m59s kube-system pod/coredns-6955765f44-5h9wz 1/1 Running 0 13m kube-system pod/coredns-6955765f44-zfnwm 1/1 Running 0 13m kube-system pod/etcd-czprapd-chenext 1/1 Running 0 13m kube-system pod/kube-apiserver-czprapd-chenext 1/1 Running 0 13m kube-system pod/kube-controller-manager-czprapd-chenext 1/1 Running 0 13m kube-system pod/kube-proxy-qb8vz 1/1 Running 0 13m kube-system pod/kube-scheduler-czprapd-chenext 1/1 Running 0 13m kube-system pod/nginx-ingress-controller-6fc5bcc8c9-5szfh 1/1 Running 0 9m9s kube-system pod/storage-provisioner 1/1 Running 0 13m

Expected behavior

Normal start of Che as before TLS was made default

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

[root@czprapd-chenext ~]# minikube version minikube version: v1.8.1 commit: cbda04cf6bbe65e987ae52bb393c10099ab62014

[root@czprapd-chenext ~]# kubectl version Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.3”, GitCommit:“06ad960bfd03b39c8310aaf92d1e7c12ce618213”, GitTreeState:“clean”, BuildDate:“2020-02-11T18:14:22Z”, GoVersion:“go1.13.6”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.3”, GitCommit:“06ad960bfd03b39c8310aaf92d1e7c12ce618213”, GitTreeState:“clean”, BuildDate:“2020-02-11T18:07:13Z”, GoVersion:“go1.13.6”, Compiler:“gc”, Platform:“linux/amd64”}

Screenshots

Installation method

• chectl
• che-operator
• minishift-addon
• I don’t know chectl server:start --multiuser --platform=minikube

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify Custom VM, CentOS 7, minikube on docker

Eclipse Che Logs

che.log che-operator.log che-devfile-registry.log keycloak.log che-plugin-registry.log postgres.log events.txt

Additional context

Started happening on our automated nigthly deployed machine, which deploys che-next from nightly builds daily. I can replicate the same scenario on different machines.

[tolusha]

@mmorhun In context of this issue it is needed to improve pre-flight chects if case of --tls flag

[mmorhun]

@filipkroupa it is still possible, but not recommended, though. One of the reasons is that some Theia functionality doesn’t work properly without TLS. So, to disable it, you should provide patch to Che custom resource:

# patch.yaml
spec:
  server:
    tlsSupport: false

chectl server:start --platform=minikube --multiuser --installer=operator --che-operator-cr-patch-yaml=patch.yaml

P.S. Thank you for pointing out to misleading tls flag description. I’ll update it.