Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Failed to start a Che pod on minikube (timeout)
See original GitHub issueDescribe the bug
Command server:deploy failed with the error Failed to start a pod (Timeout) in Eclipse Che Server pod bootstrap Step.
Che version
next (development version)
Steps to reproduce
$ minikube start \
--driver=docker \
--memory=8192 \
--cpus=4 \
--vm=true \
--addons=ingress \
--container-runtime=containerd \
--kubernetes-version=v1.25.3
😄 minikube v1.28.0 on Ubuntu 22.04 (kvm/amd64)
✨ Using the docker driver based on user configuration
📌 Using rootless Docker driver
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
💾 Downloading Kubernetes v1.25.3 preload ...
> preloaded-images-k8s-v18-v1...: 638.52 MiB / 638.52 MiB 100.00% 8.49 Mi
> gcr.io/k8s-minikube/kicbase: 386.27 MiB / 386.27 MiB 100.00% 4.02 MiB p
> gcr.io/k8s-minikube/kicbase: 0 B [_________________________] ?% ? p/s 0s
> index.docker.io/kicbase/sta...: 386.27 MiB / 386.27 MiB 100.00% 5.88 Mi
> index.docker.io/kicbase/sta...: 0 B [______________________] ?% ? p/s 0s
> gcr.io/k8s-minikube/kicbase...: 386.27 MiB / 386.27 MiB 100.00% 6.95 Mi
> gcr.io/k8s-minikube/kicbase...: 0 B [______________________] ?% ? p/s 0s
> index.docker.io/kicbase/sta...: 386.27 MiB / 386.27 MiB 100.00% 9.00 Mi
> index.docker.io/kicbase/sta...: 0 B [______________________] ?% ? p/s 0s
❗ minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.36, but successfully downloaded docker.io/kicbase/stable:v0.0.36 as a fallback image
E1215 02:26:08.332347 3986 cache.go:203] Error downloading kic artifacts: failed to download kic base image or any fallback image
🔥 Creating docker container (CPUs=4, Memory=8192MB) ...
📦 Preparing Kubernetes v1.25.3 on containerd 1.6.9 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎 Verifying ingress addon...
🌟 Enabled addons: storage-provisioner, default-storageclass, ingress
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ minikube kubectl -- get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-admission-create-ztk9z 0/1 Completed 0 2m9s
ingress-nginx ingress-nginx-admission-patch-87978 0/1 Completed 1 2m9s
ingress-nginx ingress-nginx-controller-5959f988fd-hq25p 1/1 Running 0 2m9s
kube-system coredns-565d847f94-sxtxb 1/1 Running 0 2m10s
kube-system etcd-minikube 1/1 Running 0 2m20s
kube-system kindnet-dlp4w 1/1 Running 0 2m10s
kube-system kube-apiserver-minikube 1/1 Running 0 2m23s
kube-system kube-controller-manager-minikube 1/1 Running 0 2m20s
kube-system kube-proxy-pk5v8 1/1 Running 0 2m10s
kube-system kube-scheduler-minikube 1/1 Running 0 2m20s
kube-system storage-provisioner 1/1 Running 0 2m11s
chectl server:deploy \
--platform minikube \
--skip-oidc-provider-check
Enable CLI usage data to be sent to Red Hat online services. More info: https://developers.redhat.com/article/tool-data-collection [y/n]: n
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API...[1.25]
✔ Minikube preflight checklist
✔ Verify if kubectl is installed...[OK]
✔ Verify if minikube is installed...[OK]
✔ Verify if minikube is running...[OK]
✔ Enable minikube ingress addon...[Enabled]
✔ Retrieving minikube IP and domain for ingress URLs...[192.168.49.2.nip.io]
✔ Checking minikube version...[1.28.0]
✔ Create Namespace eclipse-che...[Created]
✔ Install Cert Manager v1.8.2
✔ Apply resources...[Created]
✔ Wait for Cert Manager pods ready...[OK]
✔ Install Dex
✔ Create Namespace dex...[Created]
✔ Create Certificates...[Created: /tmp/dex-ca.crt]
✔ Create ConfigMap dex-ca...[Created]
✔ Create ServiceAccount dex...[Created]
✔ Create ClusterRole dex...[Created]
✔ Create ClusterRoleBinding dex...[Created]
✔ Create Service dex...[Created]
✔ Create Ingress dex...[Created]
✔ Generate Dex username and password...[OK: admin:admin]
✔ Create ConfigMap dex...[Created]
✔ Create Deployment dex...[Created]
✔ Configure API server
✔ Create /etc/ca-certificates directory...[Created]
✔ Copy Dex certificate into Minikube...[OK]
✔ Configure Minikube API server...[OK]
✔ Wait for Minikube API server...[OK]
✔ Start following Eclipse Che installation logs...[OK]
✔ Deploy Eclipse Che operator
✔ Install Dev Workspace operator
✔ Create Namespace devworkspace-controller...[Created]
✔ Create Dev Workspace operator resources...[Created]
✔ Wait for Dev Workspace operator ready...[OK]
✔ Create DevWorkspaceOperatorConfig devworkspace-operator-config...[Created]
✔ Create ServiceAccount che-operator...[Created]
✔ Create RBAC
✔ Create Role che-operator-leader-election...[Created]
✔ Create Role che-operator...[Created]
✔ Create RoleBinding che-operator-leader-election...[Created]
✔ Create RoleBinding che-operator...[Created]
✔ Create RoleBinding eclipse-che-che-operator...[Created]
✔ Create RoleBinding eclipse-che-che-operator...[Created]
✔ Wait for Cert Manager pods ready...[OK]
✔ Create Certificate che-operator-serving-cert...[Created]
✔ Create Issuer che-operator-selfsigned-issuer...[Created]
✔ Create Service che-operator-service...[Created]
✔ Create CRD checlusters.org.eclipse.che...[Created]
✔ Waiting...[OK]
✔ Create Deployment che-operator...[Created]
✔ Eclipse Che Operator pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Create ValidatingWebhookConfiguration org.eclipse.che...[Created]
✔ Create MutatingWebhookConfiguration org.eclipse.che...[Created]
✔ Create CheCluster Custom Resource...[Created]
❯ Wait for Eclipse Che ready
✔ Postgres pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Devfile Registry pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Plugin Registry pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Dashboard pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Gateway pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
❯ Eclipse Che Server pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✖ Starting
→ ]
Wait Eclipse Che active
Retrieving Eclipse Che self-signed CA certificate
Prepare post installation output
Error: Command server:deploy failed with the error: Failed to start a pod: Timeout:
Pod: che-5bf947d9b9-chrzp
status: [
{
"containerID": "containerd://ef8866d2e1888d55e6a2e449279fa0f3418fab797d4fc433d9aa21fd1536627b",
"image": "quay.io/eclipse/che-server:next",
"imageID": "quay.io/eclipse/che-server@sha256:1948b66650757364858b7405e7fb3e6ab885b6414108995ff720d8220c39d9c8",
"lastState": {},
"name": "che",
"ready": false,
"restartCount": 0,
"started": true,
"state": {
"running": {
"startedAt": "2022-12-15T02:51:53.000Z"
}
}
}
]
conditions: [
{
"lastProbeTime": null,
"lastTransitionTime": "2022-12-15T02:50:27.000Z",
"status": "True",
"type": "Initialized"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2022-12-15T02:50:27.000Z",
"message": "containers with unready status: [che]",
"reason": "ContainersNotReady",
"status": "False",
"type": "Ready"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2022-12-15T02:50:27.000Z",
"message": "containers with unready status: [che]",
"reason": "ContainersNotReady",
"status": "False",
"type": "ContainersReady"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2022-12-15T02:50:27.000Z",
"status": "True",
"type": "PodScheduled"
}
] See details: /home/lima.linux/.cache/chectl/error.log. Eclipse Che logs: /tmp/chectl-logs/1671071594939.
$ minikube kubectl -- get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-7778d64785-8pjq2 1/1 Running 1 (23m ago) 26m
cert-manager cert-manager-cainjector-5c7b85f464-wbf52 1/1 Running 1 (22m ago) 26m
cert-manager cert-manager-webhook-58b97ccf69-w2cjm 1/1 Running 1 (22m ago) 26m
devworkspace-controller devworkspace-controller-manager-55fc7f998f-cq4wg 2/2 Running 0 21m
devworkspace-controller devworkspace-webhook-server-77bbd64b4c-p49vh 2/2 Running 0 21m
dex dex-74bb9f7ddd-xq8gq 1/1 Running 1 (22m ago) 25m
eclipse-che che-5bf947d9b9-chrzp 0/1 Running 1 (44s ago) 9m45s
eclipse-che che-dashboard-59f6ddf747-j6df2 1/1 Running 0 10m
eclipse-che che-gateway-5554f9d575-fvmtk 3/4 CrashLoopBackOff 5 (22s ago) 9m46s
eclipse-che che-operator-6664d7cdf6-pkfm8 1/1 Running 0 20m
eclipse-che che-tls-job-2gjtv 0/1 Completed 0 19m
eclipse-che devfile-registry-df84c9484-f7gb8 1/1 Running 0 18m
eclipse-che plugin-registry-576cf48d78-x8x7m 1/1 Running 0 17m
eclipse-che postgres-845dc7b567-q4qfb 1/1 Running 0 19m
ingress-nginx ingress-nginx-admission-create-ztk9z 0/1 Completed 0 30m
ingress-nginx ingress-nginx-admission-patch-87978 0/1 Completed 1 30m
ingress-nginx ingress-nginx-controller-5959f988fd-hq25p 1/1 Running 1 (22m ago) 30m
kube-system coredns-565d847f94-sxtxb 1/1 Running 1 (23m ago) 30m
kube-system etcd-minikube 1/1 Running 2 30m
kube-system kindnet-dlp4w 1/1 Running 1 (23m ago) 30m
kube-system kube-apiserver-minikube 1/1 Running 2 (23m ago) 23m
kube-system kube-controller-manager-minikube 1/1 Running 3 (22m ago) 30m
kube-system kube-proxy-pk5v8 1/1 Running 1 (23m ago) 30m
kube-system kube-scheduler-minikube 1/1 Running 1 (23m ago) 30m
kube-system storage-provisioner 1/1 Running 1 (23m ago) 30m
Expected behavior
chectl server:deploy is success.
Runtime
minikube
Screenshots
No response
Installation method
chectl/next
Environment
other (please specify in additional context)
Eclipse Che Logs
$cat events.txt
LAST SEEN TYPE REASON OBJECT MESSAGE
0s Normal Issuing certificate/che-operator-serving-cert Issuing certificate as Secret does not exist
1s Normal Generated certificate/che-operator-serving-cert Stored new private key in temporary Secret resource "che-operator-serving-cert-rgknw"
0s Normal Requested certificate/che-operator-serving-cert Created new CertificateRequest resource "che-operator-serving-cert-gtbbz"
0s Normal cert-manager.io certificaterequest/che-operator-serving-cert-gtbbz Certificate request has been approved by cert-manager.io
0s Warning BadConfig certificaterequest/che-operator-serving-cert-gtbbz Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
0s Normal CertificateIssued certificaterequest/che-operator-serving-cert-gtbbz Certificate fetched from issuer successfully
0s Normal Issuing certificate/che-operator-serving-cert The certificate has been successfully issued
0s Normal ScalingReplicaSet deployment/che-operator Scaled up replica set che-operator-6664d7cdf6 to 1
0s Normal SuccessfulCreate replicaset/che-operator-6664d7cdf6 Created pod: che-operator-6664d7cdf6-pkfm8
0s Normal Scheduled pod/che-operator-6664d7cdf6-pkfm8 Successfully assigned eclipse-che/che-operator-6664d7cdf6-pkfm8 to minikube
0s Normal Pulling pod/che-operator-6664d7cdf6-pkfm8 Pulling image "quay.io/eclipse/che-operator:next"
0s Normal Pulled pod/che-operator-6664d7cdf6-pkfm8 Successfully pulled image "quay.io/eclipse/che-operator:next" in 12.255411894s
0s Normal Created pod/che-operator-6664d7cdf6-pkfm8 Created container che-operator
0s Normal Started pod/che-operator-6664d7cdf6-pkfm8 Started container che-operator
0s Normal LeaderElection configmap/e79b08a4.org.eclipse.che che-operator-6664d7cdf6-pkfm8_c61f7241-b04c-427a-928b-f5df7f331623 became leader
0s Normal LeaderElection lease/e79b08a4.org.eclipse.che che-operator-6664d7cdf6-pkfm8_c61f7241-b04c-427a-928b-f5df7f331623 became leader
0s Normal SuccessfulCreate job/che-tls-job Created pod: che-tls-job-2gjtv
0s Normal Scheduled pod/che-tls-job-2gjtv Successfully assigned eclipse-che/che-tls-job-2gjtv to minikube
0s Normal Pulling pod/che-tls-job-2gjtv Pulling image "quay.io/eclipse/che-tls-secret-creator:alpine-01a4c34"
0s Normal Pulled pod/che-tls-job-2gjtv Successfully pulled image "quay.io/eclipse/che-tls-secret-creator:alpine-01a4c34" in 10.749946342s
0s Normal Created pod/che-tls-job-2gjtv Created container che-tls-job-job-container
0s Normal Started pod/che-tls-job-2gjtv Started container che-tls-job-job-container
0s Normal Completed job/che-tls-job Job completed
0s Normal Sync ingress/che Scheduled for sync
0s Normal ScalingReplicaSet deployment/postgres Scaled up replica set postgres-845dc7b567 to 1
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
0s Normal Provisioning persistentvolumeclaim/postgres-data External provisioner is provisioning volume for claim "eclipse-che/postgres-data"
0s Normal ProvisioningSucceeded persistentvolumeclaim/postgres-data Successfully provisioned volume pvc-a7ebe877-336e-4081-a0de-5990377d5382
0s Normal SuccessfulCreate replicaset/postgres-845dc7b567 Created pod: postgres-845dc7b567-q4qfb
0s Warning FailedScheduling pod/postgres-845dc7b567-q4qfb 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
0s Normal Scheduled pod/postgres-845dc7b567-q4qfb Successfully assigned eclipse-che/postgres-845dc7b567-q4qfb to minikube
0s Normal Pulling pod/postgres-845dc7b567-q4qfb Pulling image "quay.io/eclipse/che--centos--postgresql-13-centos7:1-71b24684d64da46f960682cc4216222a7e4ed8b1a31dd5a865b3e71afdea20d2"
1s Normal Sync ingress/che Scheduled for sync
0s Normal Pulled pod/postgres-845dc7b567-q4qfb Successfully pulled image "quay.io/eclipse/che--centos--postgresql-13-centos7:1-71b24684d64da46f960682cc4216222a7e4ed8b1a31dd5a865b3e71afdea20d2" in 49.758381736s
0s Normal Created pod/postgres-845dc7b567-q4qfb Created container postgres
1s Normal Started pod/postgres-845dc7b567-q4qfb Started container postgres
0s Normal ScalingReplicaSet deployment/devfile-registry Scaled up replica set devfile-registry-df84c9484 to 1
0s Normal SuccessfulCreate replicaset/devfile-registry-df84c9484 Created pod: devfile-registry-df84c9484-f7gb8
0s Normal Scheduled pod/devfile-registry-df84c9484-f7gb8 Successfully assigned eclipse-che/devfile-registry-df84c9484-f7gb8 to minikube
0s Normal Pulling pod/devfile-registry-df84c9484-f7gb8 Pulling image "quay.io/eclipse/che-devfile-registry:next"
0s Normal Pulled pod/devfile-registry-df84c9484-f7gb8 Successfully pulled image "quay.io/eclipse/che-devfile-registry:next" in 19.260568989s
0s Normal Created pod/devfile-registry-df84c9484-f7gb8 Created container devfile-registry
0s Normal Started pod/devfile-registry-df84c9484-f7gb8 Started container devfile-registry
1s Normal ScalingReplicaSet deployment/plugin-registry Scaled up replica set plugin-registry-576cf48d78 to 1
0s Normal SuccessfulCreate replicaset/plugin-registry-576cf48d78 Created pod: plugin-registry-576cf48d78-x8x7m
0s Normal Scheduled pod/plugin-registry-576cf48d78-x8x7m Successfully assigned eclipse-che/plugin-registry-576cf48d78-x8x7m to minikube
0s Normal Pulling pod/plugin-registry-576cf48d78-x8x7m Pulling image "quay.io/eclipse/che-plugin-registry:next"
0s Normal Pulled pod/plugin-registry-576cf48d78-x8x7m Successfully pulled image "quay.io/eclipse/che-plugin-registry:next" in 6m37.69084016s
0s Normal Created pod/plugin-registry-576cf48d78-x8x7m Created container plugin-registry
0s Normal Started pod/plugin-registry-576cf48d78-x8x7m Started container plugin-registry
0s Warning Unhealthy pod/plugin-registry-576cf48d78-x8x7m Readiness probe failed: Get "http://10.244.0.22:8080/plugins/": dial tcp 10.244.0.22:8080: connect: connection refused
0s Normal ScalingReplicaSet deployment/che-dashboard Scaled up replica set che-dashboard-59f6ddf747 to 1
0s Normal SuccessfulCreate replicaset/che-dashboard-59f6ddf747 Created pod: che-dashboard-59f6ddf747-j6df2
0s Normal Scheduled pod/che-dashboard-59f6ddf747-j6df2 Successfully assigned eclipse-che/che-dashboard-59f6ddf747-j6df2 to minikube
1s Normal Pulling pod/che-dashboard-59f6ddf747-j6df2 Pulling image "quay.io/eclipse/che-dashboard:next"
0s Normal Pulled pod/che-dashboard-59f6ddf747-j6df2 Successfully pulled image "quay.io/eclipse/che-dashboard:next" in 21.210768445s
0s Normal Created pod/che-dashboard-59f6ddf747-j6df2 Created container che-dashboard
0s Normal Started pod/che-dashboard-59f6ddf747-j6df2 Started container che-dashboard
0s Warning Unhealthy pod/che-dashboard-59f6ddf747-j6df2 Readiness probe failed: Get "http://10.244.0.23:8080/": dial tcp 10.244.0.23:8080: connect: connection refused
0s Warning Unhealthy pod/che-dashboard-59f6ddf747-j6df2 Readiness probe failed: Get "http://10.244.0.23:8080/": dial tcp 10.244.0.23:8080: connect: connection refused
0s Warning Unhealthy pod/che-dashboard-59f6ddf747-j6df2 Readiness probe failed: Get "http://10.244.0.23:8080/": dial tcp 10.244.0.23:8080: connect: connection refused
0s Normal ScalingReplicaSet deployment/che-gateway Scaled up replica set che-gateway-5554f9d575 to 1
0s Normal SuccessfulCreate replicaset/che-gateway-5554f9d575 Created pod: che-gateway-5554f9d575-fvmtk
0s Normal Scheduled pod/che-gateway-5554f9d575-fvmtk Successfully assigned eclipse-che/che-gateway-5554f9d575-fvmtk to minikube
0s Normal ScalingReplicaSet deployment/che Scaled up replica set che-5bf947d9b9 to 1
0s Normal SuccessfulCreate replicaset/che-5bf947d9b9 Created pod: che-5bf947d9b9-chrzp
0s Normal Scheduled pod/che-5bf947d9b9-chrzp Successfully assigned eclipse-che/che-5bf947d9b9-chrzp to minikube
0s Normal Pulling pod/che-5bf947d9b9-chrzp Pulling image "quay.io/eclipse/che-server:next"
0s Normal Pulling pod/che-gateway-5554f9d575-fvmtk Pulling image "quay.io/eclipse/che--traefik:v2.8.1-4e52a5e2495484f5e19a49edfd2f652b0bce7b3603fa0df545ed90168ffae1c3"
0s Normal Pulled pod/che-5bf947d9b9-chrzp Successfully pulled image "quay.io/eclipse/che-server:next" in 1m19.435363286s
1s Normal Created pod/che-5bf947d9b9-chrzp Created container che
0s Normal Started pod/che-5bf947d9b9-chrzp Started container che
0s Normal Pulled pod/che-gateway-5554f9d575-fvmtk Successfully pulled image "quay.io/eclipse/che--traefik:v2.8.1-4e52a5e2495484f5e19a49edfd2f652b0bce7b3603fa0df545ed90168ffae1c3" in 1m44.208662947s
0s Normal Created pod/che-gateway-5554f9d575-fvmtk Created container gateway
0s Normal Started pod/che-gateway-5554f9d575-fvmtk Started container gateway
0s Normal Pulling pod/che-gateway-5554f9d575-fvmtk Pulling image "quay.io/che-incubator/configbump:0.1.4"
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": dial tcp 10.244.0.25:8080: connect: connection refused
0s Normal Pulled pod/che-gateway-5554f9d575-fvmtk Successfully pulled image "quay.io/che-incubator/configbump:0.1.4" in 13.608099796s
0s Normal Created pod/che-gateway-5554f9d575-fvmtk Created container configbump
0s Normal Started pod/che-gateway-5554f9d575-fvmtk Started container configbump
0s Normal Pulling pod/che-gateway-5554f9d575-fvmtk Pulling image "quay.io/oauth2-proxy/oauth2-proxy:v7.4.0"
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": dial tcp 10.244.0.25:8080: connect: connection refused
0s Normal Pulled pod/che-gateway-5554f9d575-fvmtk Successfully pulled image "quay.io/oauth2-proxy/oauth2-proxy:v7.4.0" in 12.354477867s
0s Normal Created pod/che-gateway-5554f9d575-fvmtk Created container oauth-proxy
0s Normal Started pod/che-gateway-5554f9d575-fvmtk Started container oauth-proxy
0s Normal Pulling pod/che-gateway-5554f9d575-fvmtk Pulling image "quay.io/brancz/kube-rbac-proxy:v0.11.0"
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": dial tcp 10.244.0.25:8080: connect: connection refused
0s Normal Pulled pod/che-gateway-5554f9d575-fvmtk Successfully pulled image "quay.io/brancz/kube-rbac-proxy:v0.11.0" in 14.318655735s
0s Normal Created pod/che-gateway-5554f9d575-fvmtk Created container kube-rbac-proxy
0s Normal Started pod/che-gateway-5554f9d575-fvmtk Started container kube-rbac-proxy
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Normal Pulled pod/che-gateway-5554f9d575-fvmtk Container image "quay.io/oauth2-proxy/oauth2-proxy:v7.4.0" already present on machine
0s Normal Created pod/che-gateway-5554f9d575-fvmtk Created container oauth-proxy
0s Normal Started pod/che-gateway-5554f9d575-fvmtk Started container oauth-proxy
0s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning BackOff pod/che-gateway-5554f9d575-fvmtk Back-off restarting failed container
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Normal Pulled pod/che-gateway-5554f9d575-fvmtk Container image "quay.io/oauth2-proxy/oauth2-proxy:v7.4.0" already present on machine
0s Normal Created pod/che-gateway-5554f9d575-fvmtk Created container oauth-proxy
0s Normal Started pod/che-gateway-5554f9d575-fvmtk Started container oauth-proxy
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Warning BackOff pod/che-gateway-5554f9d575-fvmtk Back-off restarting failed container
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Warning BackOff pod/che-gateway-5554f9d575-fvmtk Back-off restarting failed container
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Normal Pulled pod/che-gateway-5554f9d575-fvmtk Container image "quay.io/oauth2-proxy/oauth2-proxy:v7.4.0" already present on machine
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
1s Warning Unhealthy pod/che-5bf947d9b9-chrzp Readiness probe failed: Get "http://10.244.0.25:8080/api/system/state": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s Warning BackOff pod/che-gateway-5554f9d575-fvmtk Back-off restarting failed container
$ cat che-gateway-5554f9d575-fvmtk/gateway.log
time="2022-12-15T02:52:20Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
time="2022-12-15T02:52:20Z" level=info msg="Traefik version 2.8.1 built on 2022-07-11T14:06:57Z"
time="2022-12-15T02:52:20Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2022-12-15T02:52:20Z" level=warning msg="Traefik Pilot is deprecated and will be removed soon. Please check our Blog for migration instructions later this year."
time="2022-12-15T02:52:20Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2022-12-15T02:52:20Z" level=info msg="Starting provider *file.Provider"
time="2022-12-15T02:52:20Z" level=info msg="Starting provider *traefik.Provider"
time="2022-12-15T02:52:20Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
$ cat che-gateway-5554f9d575-fvmtk/kube-rbac-proxy.log
I1215 02:53:04.427320 1 main.go:150] Reading config file: /etc/kube-rbac-proxy/authorization-config.yaml
I1215 02:53:04.509283 1 main.go:180] Valid token audiences:
I1215 02:53:04.538301 1 main.go:379] Listening insecurely on 0.0.0.0:8089
$ cat che-gateway-5554f9d575-fvmtk/oauth-proxy.log
[2022/12/15 02:52:47] [provider.go:55] Performing OIDC Discovery...
[2022/12/15 02:53:28] [main.go:60] ERROR: Failed to initialise OAuth2 Proxy: error intiailising provider: could not create provider data: error building OIDC ProviderVerifier: could not get verifier builder: error while discovery OIDC configuration: failed to discover OIDC configuration: error performing request: Get "http://dex.dex:5556/.well-known/openid-configuration": dial tcp: lookup dex.dex on 10.96.0.10:53: read udp 10.244.0.24:50335->10.96.0.10:53: i/o timeout
$ cat che-gateway-5554f9d575-fvmtk/configbump.log
{"level":"info","ts":1671072755.075611,"logger":"controller-runtime.controller","caller":"zapr@v0.1.0/zapr.go:69","msg":"Starting EventSource","controller":"config-bump","source":"kind source: /, Kind="}
{"level":"info","ts":1671072755.1898997,"logger":"controller-runtime.controller","caller":"zapr@v0.1.0/zapr.go:69","msg":"Starting Controller","controller":"config-bump"}
{"level":"info","ts":1671072755.310251,"logger":"controller-runtime.controller","caller":"zapr@v0.1.0/zapr.go:69","msg":"Starting workers","controller":"config-bump","worker count":1}
$ cat che-5bf947d9b9-chrzp/che.log
Using custom assembly from /home/user/eclipse-che
Found a custom cert. Adding it to java trust store /home/user/cacerts based on /usr/lib/jvm/jre/lib/security/cacerts
Trust this certificate? [no]: Certificate was added to keystore
Certificate was added to keystore
Imported 1 certificates from /public-certs/dex-ca.ca.crt
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
15-Dec-2022 02:52:39.898 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/10.0.14
15-Dec-2022 02:52:40.077 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Dec 2 2021 22:01:36 UTC
15-Dec-2022 02:52:40.078 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.14.0
15-Dec-2022 02:52:40.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
15-Dec-2022 02:52:40.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.15.0-53-generic
15-Dec-2022 02:52:40.080 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
15-Dec-2022 02:52:40.081 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el8_6.x86_64
15-Dec-2022 02:52:40.082 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.17+8-LTS
15-Dec-2022 02:52:40.083 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Red Hat, Inc.
15-Dec-2022 02:52:40.084 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /home/user/eclipse-che/tomcat
15-Dec-2022 02:52:40.086 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /home/user/eclipse-che/tomcat
15-Dec-2022 02:52:40.687 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
15-Dec-2022 02:52:40.688 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
15-Dec-2022 02:52:40.688 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
15-Dec-2022 02:52:40.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
15-Dec-2022 02:52:40.720 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
15-Dec-2022 02:52:40.762 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che/tomcat/conf/logging.properties
15-Dec-2022 02:52:40.763 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
15-Dec-2022 02:52:40.764 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMPercentage=85.0
15-Dec-2022 02:52:40.764 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
15-Dec-2022 02:52:40.764 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStore=/home/user/cacerts
15-Dec-2022 02:52:40.767 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStorePassword=changeit
15-Dec-2022 02:52:40.767 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
15-Dec-2022 02:52:40.768 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che
15-Dec-2022 02:52:40.781 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/logs/
15-Dec-2022 02:52:40.782 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
15-Dec-2022 02:52:40.783 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che/tomcat/conf/tomcat-logger.xml
15-Dec-2022 02:52:40.783 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
15-Dec-2022 02:52:40.784 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
15-Dec-2022 02:52:40.784 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
15-Dec-2022 02:52:40.785 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/home/user/eclipse-che/tomcat/conf/
15-Dec-2022 02:52:40.786 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
15-Dec-2022 02:52:40.786 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che/tomcat
15-Dec-2022 02:52:40.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che/tomcat
15-Dec-2022 02:52:40.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che/tomcat/temp
15-Dec-2022 02:52:50.269 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
15-Dec-2022 02:52:50.982 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [17327] milliseconds
15-Dec-2022 02:52:52.272 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
15-Dec-2022 02:52:52.273 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.0.14]
15-Dec-2022 02:52:53.133 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
15-Dec-2022 02:53:07.984 WARNING [main] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [308] milliseconds.
15-Dec-2022 02:53:08.685 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [15,554] ms
15-Dec-2022 02:53:08.770 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
15-Dec-2022 02:53:43.585 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [34,814] ms
15-Dec-2022 02:53:43.690 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2022-12-15 02:54:33,228[main] [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63] - This=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory@3613d6a6 obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@2143563e environment={}
2022-12-15 02:54:40,341[main] [INFO ] [.e.c.a.d.WsMasterServletModule 52] - Running in native-user mode ...
Additional context
Lima(v0.13.0) VM on M1 Mac
$ uname -m
x86_64
$ cat /etc/issue
Ubuntu 22.04.1 LTS \n \l
$ docker -v
Docker version 20.10.21, build baeda1f
$ minikube version
minikube version: v1.28.0
commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f
$ kubectl version --short
Client Version: v1.25.3
Kustomize Version: v4.5.7
$ chectl version
chectl/0.0.20221214-next.5fdabb1 linux-x64 node-v16.13.2
Issue Analytics
- State:
- Created 9 months ago
- Comments:5 (1 by maintainers)
Top Results From Across the Web
Che-server fails to bootstrap on minikube #15069
Describe the bug Che7 deployment on my local machine failes at step "Che Pod Bootstrap" when executing: chectl server:start --platform ...
Read more >Unable to bring up Eclipse che on Kubernetes
I got Kubernetes installed on Ubuntu and am trying to run chectl server:start but it is failing. What am doing wrong? Below is...
Read more >Running Kubernetes Locally via Minikube - 2021
Ensures availability and scalability. It handles replication and scaling by running a specified number of copies of a pod across the cluster. It...
Read more >Error installing eclipse che using chectl - FailedScheduling
I've been trying to install Eclipse Che with minikube for a while and ... Timeout set to pod wait timeout 300000 › Installation...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Some communication issue with dex. Unfortunately I can’t reproduce locally.
@scarint thank you for the advice. I will try it.