pod has unbound immediate PersistentVolumeClaims

Describe the bug

Times out when scheduling/downloading images. Logs indicate timeout waiting for postgres volumes to be bound

Che version latest

Steps to reproduce

ran chectl server:start --platform k8s --multiuser --domain=che.test.maap --k8spodwaittimeout=300000 Expected behavior

Eclipse che is deployed and a url is generated Runtime environment: Kubernetes

Kubernetes cluster: 1.13

kubectl version Client Version: version.Info{Major:“1”, Minor:“18”, GitVersion:“v1.18.0”, GitCommit:“9e991415386e4cf155a24b1da15becaa390438d8”, GitTreeState:“clean”, BuildDate:“2020-03-25T14:58:59Z”, GoVersion:“go1.13.8”, Compiler:“gc”, Platform:“windows/amd64”} Unable to connect to the server: dial tcp 90.84.245.177:5443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Screenshots

 chectl server:start --platform k8s --multiuser --domain=che.test.maap --k8spodwaittimeout=300000
› Current Kubernetes context: 'external'
  √ Verify Kubernetes API...OK
  √ �  Looking for an already existing Eclipse Che instance
    √ Verify if Eclipse Che is deployed into namespace "che"...it is not
  √ ✈️  Kubernetes preflight checklist
    √ Verify if kubectl is installed
    √ Verify remote kubernetes status...done.
    √ Check Kubernetes version: Found v1.13.10-r0-CCE2.0.26.B001.
    √ Verify domain is set...set to che.test.maap.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in 'C:\Users\tkossoko\AppData\Local\Temp\1\chectl-logs\1597220181106'
  √ Start following logs
    √ Start following Operator logs...done
    √ Start following Eclipse Che logs...done
    √ Start following Postgres logs...done
    √ Start following Keycloak logs...done
    √ Start following Plugin registry logs...done
    √ Start following Devfile registry logs...done
  √ Start following events
    √ Start following namespace events...done
  √ �‍  Running the Eclipse Che operator
    √ Copying operator resources...done.
    √ Create Namespace (che)...It already exists.
    √ Create ServiceAccount che-operator in namespace che...It already exists.
    √ Create Role che-operator in namespace che...It already exists.
    √ Create ClusterRole che-operator...It already exists.
    √ Create RoleBinding che-operator in namespace che...It already exists.
    √ Create ClusterRoleBinding che-operator...It already exists.
    √ Create CRD checlusters.org.eclipse.che...It already exists.
    √ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    √ Create deployment che-operator in namespace che...It already exists.
    √ Create the Custom Resource of type checlusters.org.eclipse.che in the namespace che...It already exists.
  > ✅  Post installation checklist
    > Eclipse Che pod bootstrap
      × scheduling
        → ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
        downloading images
        starting
      Retrieving Eclipse Che server URL
      Eclipse Che status check
    Retrieving Che self-signed CA certificate
 »   Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
 »   Installation failed, check logs in 'C:\Users\tkossoko\AppData\Local\Temp\1\chectl-logs\1597220181106'

Logs :

LAST SEEN   TYPE      REASON             KIND                    MESSAGE
45m         Warning   FailedScheduling   Pod                     pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
21s         Warning   FailedScheduling   Pod                     pod has unbound immediate PersistentVolumeClaims (repeated 4 times)
55s         Normal    FailedBinding      PersistentVolumeClaim   no persistent volumes available for this claim and no storage class is set
NAME                             AGE
postgres-data.162a39d1dba1091a   19h
postgres-6f55cd44c9-tht7b.162a757f8680b90f   45m
postgres-data.162a39d1dba1091a               19h
postgres-6f55cd44c9-tht7b.162a757f8680b90f   50m

time="2020-08-11T13:29:39Z" level=info msg="Default 'info' log level is applied"
time="2020-08-11T13:29:39Z" level=info msg="Go Version: go1.13.4"
time="2020-08-11T13:29:39Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-08-11T13:29:39Z" level=info msg="operator-sdk Version: v0.5.0"
time="2020-08-11T13:29:39Z" level=info msg="Operator is running on Kubernetes"
time="2020-08-11T13:29:40Z" level=info msg="Registering Che Components Types"
time="2020-08-11T13:29:40Z" level=info msg="Starting the Cmd"
time="2020-08-11T13:29:40Z" level=info msg="Creating a new object: ServiceAccount, name che-tls-job-service-account"
time="2020-08-11T13:29:40Z" level=info msg="Creating a new object: Role, name che-tls-job-role"
time="2020-08-11T13:29:40Z" level=info msg="Creating a new object: RoleBinding, name che-tls-job-role-binding"
time="2020-08-11T13:29:40Z" level=info msg="Creating a new object: Job, name che-tls-job"
time="2020-08-11T13:29:40Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:41Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:42Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:43Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:44Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:45Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:46Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:47Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:48Z" level=info msg="Waiting on job 'che-tls-job' to be finished"
time="2020-08-11T13:29:50Z" level=info msg="Import public part of Eclipse Che self-signed CA certificvate from \"self-signed-certificate\" secret into your browser."
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: ServiceAccount, name che"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on service account 'che' to be created"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: ServiceAccount, name che-workspace"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on service account 'che-workspace' to be created"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: Role, name exec"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on role 'exec' to be created"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: Role, name view"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on role 'view' to be created"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: RoleBinding, name che"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on role binding 'che' to be created"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: RoleBinding, name che-workspace-exec"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on role binding 'che-workspace-exec' to be created"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: RoleBinding, name che-workspace-view"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on role binding 'che-workspace-view' to be created"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with installation flavor: che"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: Secret, name che-postgres-secret"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Postgres Secret: che-postgres-secret"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: Secret, name che-identity-postgres-secret"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Identity Provider Postgres Secret: che-identity-postgres-secret"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: Secret, name che-identity-secret"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Identity Provider Secret: che-identity-secret"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Postgres DB: dbche"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Postgres hostname: postgres"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Postgres port: 5432"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Keycloak realm: che"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with Keycloak client ID: che-public"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Updating eclipse-che CR with log level: INFO"
time="2020-08-11T13:29:51Z" level=info msg="Custom resource eclipse-che updated"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: Service, name postgres"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on service 'postgres' to be ready"
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: PersistentVolumeClaim, name postgres-data"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on pvc 'postgres-data' to be bound. Sometimes PVC can be bound only when the first consumer is created."
time="2020-08-11T13:29:51Z" level=info msg="Creating a new object: Deployment, name postgres"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on deployment 'postgres' to be ready"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on deployment 'postgres' to be ready"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on deployment 'postgres' to be ready"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on deployment 'postgres' to be ready"
time="2020-08-11T13:29:51Z" level=info msg="Waiting on deployment 'postgres' to be ready"
time="2020-08-11T13:39:52Z" level=info msg="Waiting on deployment 'postgres' to be ready"
time="2020-08-11T23:29:40Z" level=info msg="Waiting on deployment 'postgres' to be ready"
time="2020-08-11T23:29:40Z" level=info msg="Waiting on deployment 'postgres' to be ready"

[tolusha]

Probably some leftovers in the namespace. I don’t see storageClassName is set Could you pls remove the namespace and try again?

[KOSSOKO]

It is working thanks. Now che is downloading the images. I deleted the namespace and rerun the command

kubectl delete namespaces che
namespace "che" deleted
chectl server:start --platform k8s --multiuser --domain=che.test.maap --k8spodwaittimeout=400000 --postgres-pvc-storage-class-name="sata" --workspace-pvc-storage-class-name="sata"

Thanks a lot