Eclipse Che pod bootstrap timeout on chectl install, when using Che operator with TLS and unsigned certificate on non-OpenShift kube
See original GitHub issueDescribe the bug
When attempting to install Che 7.9.0 on generic Kubernetes, with TLS enabled and a self-signed certificate, using chectl via the Che operator, the Che pod fails to start due to an inability to connect to Keycloak.
chectl server:start --platform=k8s --installer=operator --domain=(cluster ip).nip.io --che-operator-cr-yaml=./codewind-checluster.yaml --che-operator-image=quay.io/eclipse/che-operator:7.9.0 --tls --self-signed-cert
The Che pod appears not to allow connecting to Keycloak via a self-signed certificate.
As per the attached Che pod logs, the che pod is failing to start due to the following exception
Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/.well-known/openid-configuration
at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:104)
at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings$$FastClassByGuice$$e0d0786b.newInstance(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
(... edit ...)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4699)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:970)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1841)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching keycloak-che.9.42.80.171.nip.io found.
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
(... edit ...)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
at java.net.URL.openStream(URL.java:1067)
at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:97)
... 124 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching keycloak-che.9.42.80.171.nip.io found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:462)
(... edit ...)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
... 138 more
The che pod appears to be attempting to access this URL https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/.well-known/openid-configuration
URL, which I am able to successfully access from my browser (albeit behind a self-signed cert browser warning) and curl:
jgw@pulse-orange$ curl https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/.well-known/openid-configuration --insecure
{"issuer":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che","authorization_endpoint":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/auth","token_endpoint":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/token","token_introspection_endpoint":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/token/introspect","userinfo_endpoint":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/userinfo","end_session_endpoint":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/logout","jwks_uri":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/certs","check_session_iframe":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/login-status-iframe.html","grant_types_supported":["authorization_code","implicit","refresh_token","password","client_credentials"],"response_types_supported":["code","none","id_token","token","id_token token","code id_token","code token","code id_token token"],"subject_types_supported":["public","pairwise"],"id_token_signing_alg_values_supported":["PS384","ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","PS256","PS512","RS512"],"userinfo_signing_alg_values_supported":["PS384","ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","PS256","PS512","RS512","none"],"request_object_signing_alg_values_supported":["PS384","ES384","RS384","ES256","RS256","ES512","PS256","PS512","RS512","none"],"response_modes_supported":["query","fragment","form_post"],"registration_endpoint":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/clients-registrations/openid-connect","token_endpoint_auth_methods_supported":["private_key_jwt","client_secret_basic","client_secret_post","client_secret_jwt"],"token_endpoint_auth_signing_alg_values_supported":["RS256"],"claims_supported":["aud","sub","iss","auth_time","name","given_name","family_name","preferred_username","email"],"claim_types_supported":["normal"],"claims_parameter_supported":false,"scopes_supported":["openid","microprofile-jwt","web-origins","roles","phone","address","email","profile","offline_access"],"request_parameter_supported":true,"request_uri_parameter_supported":true,"code_challenge_methods_supported":["plain","S256"],"tls_client_certificate_bound_access_tokens":true,"introspection_endpoint":"https://keycloak-che.9.42.80.171.nip.io/auth/realms/che/protocol/openid-connect/token/introspect"}
A Helm install against the same cluster, using the following install command, does not exhibit this problem:
chectl server:start --platform=k8s --installer=helm --domain=9.42.80.171.nip.io --multiuser --tls --self-signed-cert
Che version
7.9.0
Steps to reproduce
- Generate Che self-signed certs and create them as secrets in the che namespace:
export CLUSTER_IP=(cluster ip)
CA_CN=eclipse-che-signer
DOMAIN="*.$CLUSTER_IP.nip.io"
OPENSSL_CNF="/usr/lib/ssl/openssl.cnf"
OUT_DIR="`cd ~;pwd`"
openssl genrsa -out rootCA.key 4096
openssl req -x509 \
-new -nodes \
-key rootCA.key \
-sha256 \
-days 1024 \
-out rootCA.crt \
-subj /CN=${CA_CN} \
-reqexts SAN \
-extensions SAN \
-config <(cat ${OPENSSL_CNF} \
<(printf '[SAN]\nbasicConstraints=critical, CA:TRUE\nkeyUsage=keyCertSign, cRLSign, digitalSignature, keyEncipherment'))
openssl genrsa -out domain.key 2048
openssl req -new -sha256 \
-key domain.key \
-subj "/O=EclipseChe/CN=${DOMAIN}" \
-reqexts SAN \
-config <(cat ${OPENSSL_CNF} \
<(printf "\n[SAN]\nsubjectAltName=DNS:${DOMAIN}\nbasicConstraints=critical, CA:FALSE\nkeyUsage=keyCertSign, digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth")) \
-out domain.csr
openssl x509 \
-req \
-sha256 \
-extfile <(printf "subjectAltName=DNS:${DOMAIN}\nbasicConstraints=critical, CA:FALSE\nkeyUsage=keyCertSign, digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth") \
-days 365 \
-in domain.csr \
-CA rootCA.crt \
-CAkey rootCA.key \
-CAcreateserial -out "$OUT_DIR/domain.crt"
cp rootCA.crt "$OUT_DIR/ca.crt"
kubectl create namespace che
kubectl create secret tls che-tls --key=domain.key "--cert=$OUT_DIR/domain.crt" -n che
kubectl create secret generic self-signed-cert "--from-file=$OUT_DIR/ca.crt" -n che
- Apply the custom clusterrole, which will be referenced in the next step
kubectl apply -f https://raw.githubusercontent.com/eclipse/codewind-che-plugin/master/setup/install_che/codewind-clusterrole.yaml
- Download CheCluster operator resource YAML for use by chectl
wget https://raw.githubusercontent.com/eclipse/codewind-che-plugin/master/setup/install_che/che-operator/codewind-checluster.yaml
- Edit the file and replace
ingressDomain: ''
with your ingress domain (eg ` ingressDomain: β9.42.80.171.nip.ioβ)
- On a non-OpenShift Kubernetes distribution, attempt to install Che using operator install from chectl, using a self-signed certificate.
chectl server:start --platform=k8s --installer=operator --domain=(cluster ip).nip.io --che-operator-cr-yaml=./codewind-checluster.yaml --che-operator-image=quay.io/eclipse/che-operator:7.9.0 --tls --self-signed-cert
Output
chectl server:start --platform=k8s --installer=operator --domain=9.42.80.171.nip.io --che-operator-cr-yaml=/home/ibmadmin/codewind-checluster.yaml --che-operator-image=quay.io/eclipse/che-operator:7.9.0 --tls --self-signed-cert
β Verify Kubernetes API...OK
β π Looking for an already existing Eclipse Che instance
β Verify if Eclipse Che is deployed into namespace "che"...it is not
β βοΈ Kubernetes preflight checklist
β Verify if kubectl is installed
β Verify remote kubernetes status...done.
β Check Kubernetes version: Found v1.17.3+k3s1.
β Verify domain is set...set to 9.42.80.171.nip.io.
Eclipse Che logs will be available in '/tmp/chectl-logs/1583510159056'
β Start following logs
β Start following Eclipse Che logs...done
β Start following Postgres logs...done
β Start following Keycloak logs...done
β Start following Plugin registry logs...done
β Start following Devfile registry logs...done
β Start following events
β Start following namespace events...done
βΊ Warning: Eclipse Che will be deployed in Multi-User mode as 'operator' installer supports only that mode.
β πβ Running the Che Operator
β Copying operator resources...done.
β Create Namespace (che)...It already exists.
β Create ServiceAccount che-operator in namespace che...done.
β Create Role che-operator in namespace che...done.
β Create ClusterRole che-operator...done.
β Create RoleBinding che-operator in namespace che...done.
β Create ClusterRoleBinding che-operator...done.
β Create CRD checlusters.org.eclipse.che...done.
β Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
β Create deployment che-operator in namespace che...done.
β Create Eclipse Che Cluster eclipse-che in namespace che...done.
β― β
Post installation checklist
β PostgreSQL pod bootstrap
β scheduling...done.
β downloading images...done.
β starting...done.
β Keycloak pod bootstrap
β scheduling...done.
β downloading images...done.
β starting...done.
β Devfile registry pod bootstrap
β scheduling...done.
β downloading images...done.
β starting...done.
β Plugin registry pod bootstrap
β scheduling...done.
β downloading images...done.
β starting...done.
β― Eclipse Che pod bootstrap
β scheduling...done.
β downloading images...done.
β starting
β ERR_TIMEOUT: Timeout set to pod ready timeout 130000
Retrieving Eclipse Che Server URL
Eclipse Che status check
βΊ Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
βΊ Installation failed, check logs in '/tmp/chectl-logs/1583510159056'
See attached logs below.
Expected behavior
Che pod to successfully start after connecting to Keycloak endpoint.
Runtime
Kubernetes:
- Client Version: version.Info{Major:β1β, Minor:β17β, GitVersion:βv1.17.3+k3s1β, GitCommit:β5b17a175ce333dfb98cb8391afeb1f34219d9275β, GitTreeState:βcleanβ, BuildDate:β2020-02-27T07:28:53Zβ, GoVersion:βgo1.13.8β, Compiler:βgcβ, Platform:βlinux/amd64β}
- Server Version: version.Info{Major:β1β, Minor:β17β, GitVersion:βv1.17.3+k3s1β, GitCommit:β5b17a175ce333dfb98cb8391afeb1f34219d9275β, GitTreeState:βcleanβ, BuildDate:β2020-02-27T07:28:53Zβ, GoVersion:βgo1.13.8β, Compiler:βgcβ, Platform:βlinux/amd64β}
Installation method
chectl -platform=k8s --installer=operator
, see above for more info.
Environment
Ubuntu 18.04 LTS server
Eclipse Che Logs
ZIP of /tmp/chectl-logs/1583510159056
chectl-logs.zip
Issue Analytics
- State:
- Created 4 years ago
- Comments:14 (13 by maintainers)
Top GitHub Comments
@mmorhun - Looks good, with those two changes I am able to install Che as expected. π
Re: docs, itβs this page that still suggests to use
self-signed-cert
as the secret name: https://github.com/eclipse/che-docs/blame/master/src/main/pages/che-7/contributor-guide/proc_deploy-che-with-self-signed-tls-on-kubernetes.adoc#L41@jgwest with all the changes from the PRs above it should work (tested on minikube though). And donβt forget about
self-signed-certificate
secret name. P.S. The certificate generated in the steps to reproduce is not accepted by Chrome, but ok for Firefox, curl, openssl@eder-santos
One may implicitly set
k8s.tlsSecretName
toche-tls
. And, of course, the self signed certificate secret name should beself-signed-certificate
notself-signed-cert
.