Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Workspaces creation failed - Kubernates Deployment

See original GitHub issue

Description

When I try to create a workspace in Che, this seems to be initialized fine (pods are okay) but I got “bootstrapper timeout”.

Reproduction Steps

This is my deployment yml:

---
apiVersion: v1
kind: List
items:
- apiVersion: v1                
  kind: PersistentVolume        
  metadata:                     
    name: che-volume
    labels:  
      type: local            
  spec:                         
    storageClassName: manual    
    capacity:                   
      storage: 15Gi              
    accessModes:                
      - ReadWriteOnce           
    hostPath:                   
      path: "/var/lib/kubelet/chedata" 
- apiVersion: v1
  kind: ServiceAccount
  metadata:
    labels:
      app: che
    name: che
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: che
    name: che-service
  spec:
    ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 80123
    selector:
      app: che
# - apiVersion: v1
#   kind: PersistentVolumeClaim
#   metadata:
#     labels:
#       app: che
#     name: che-data-volume
#   spec:
#     storageClassName: manual
#     accessModes:
#     - ReadWriteOnce
#     resources:
#       requests:
#         storage: 2Gi
- apiVersion: v1
  kind: ConfigMap
  metadata:
    labels:
      app: che
    name: che
  data:
    CHE_HOST: "10-118-135-252.ebiz.verizon.com"
    CHE_PORT: "8080"
    CHE_API: "http://10-118-135-252.ebiz.verizon.com/api"
    CHE_WEBSOCKET_ENDPOINT:  ws://10-118-135-252.ebiz.verizon.com/api/websocket
    CHE_DEBUG_SERVER: "true"
    CHE_INFRASTRUCTURE_ACTIVE: kubernetes
    CHE_INFRA_KUBERNETES_BOOTSTRAPPER_BINARY__URL: http://10-118-135-252.ebiz.verizon.com/agent-binaries/linux_amd64/bootstrapper/bootstrapper
    CHE_INFRA_KUBERNETES_MACHINE__START__TIMEOUT__MIN: "5"
    CHE_INFRA_KUBERNETES_MASTER__URL: ""
    CHE_INFRA_KUBERNETES_OAUTH__TOKEN: ""
    CHE_INFRA_KUBERNETES_PASSWORD: ""
    CHE_INFRA_KUBERNETES_USERNAME: ""
    CHE_INFRA_KUBERNETES_NAMESPACE: "che"
    CHE_INFRA_KUBERNETES_TRUST__CERTS: "false"
    CHE_INFRA_KUBERNETES_PVC_STRATEGY: "common"
    CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: "false"
    CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER: "0"
    CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP: "0"
    CHE_LOCAL_CONF_DIR: /etc/conf
    CHE_LOGS_DIR: /data/logs
    CHE_LOG_LEVEL: "INFO"
    CHE_MULTIUSER: "false"
    CHE_OAUTH_GITHUB_CLIENTID: ""
    CHE_OAUTH_GITHUB_CLIENTSECRET: ""
    CHE_PREDEFINED_STACKS_RELOAD__ON__START: "false"
    JAVA_OPTS: "-XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Xms20m "
    CHE_WORKSPACE_AUTO_START: "false"
    CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON: '{"nginx.ingress.kubernetes.io/rewrite-target": "/","nginx.ingress.kubernetes.io/ssl-redirect": "true","nginx.ingress.kubernetes.io/proxy-connect-timeout": "3600","nginx.ingress.kubernetes.io/proxy-read-timeout": "3600"}'
    CHE_INFRA_KUBERNETES_PVC_ENABLED: "true"
    CHE_INFRA_KUBERNETES_PVC_NAME: "claim-che-workspace"
    CHE_INFRA_KUBERNETES_PVC_QUANTITY: "10Gi"
    CHE_INFRA_KUBERNETES_PVC_ACCESS__MODE: "ReadWriteOnce"
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: che-ingress
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
  spec:
    rules:
    - host: products-app.kube-dev.ebiz.verizon.com
      http:
        paths:
        - backend:
            serviceName: che-service
            servicePort: 8080
- apiVersion: apps/v1beta2
  kind: StatefulSet
  metadata:
    labels:
      app: che
    name: che
  spec:
    serviceName: che-service
    replicas: 1
    revisionHistoryLimit: 2
    selector:
      matchLabels:
        app: che
    # strategy:
    #   type: Recreate
    template:
      metadata:
        annotations:
          pod.alpha.kubernetes.io/init-containers: '[{"image":"busybox","imagePullPolicy":"IfNotPresent","name":"fmp-volume-permission","command":["chmod","777","/data"],"volumeMounts":[{"mountPath":"/data","name":"che-data-volume"}]}]'
        labels:
          app: che
      spec:
        containers:
        - env:
          - name: CHE_HOST
            valueFrom:
              configMapKeyRef:
                key: CHE_HOST
                name: che
          - name: CHE_PORT
            valueFrom:
              configMapKeyRef:
                key: CHE_PORT
                name: che
          - name: CHE_API
            valueFrom:
              configMapKeyRef:
                key: CHE_API
                name: che
          - name: CHE_WEBSOCKET_ENDPOINT
            valueFrom:
              configMapKeyRef:
                key: CHE_WEBSOCKET_ENDPOINT
                name: che
          - name: CHE_DEBUG_SERVER
            valueFrom:
              configMapKeyRef:
                key: CHE_DEBUG_SERVER
                name: che
          - name: CHE_INFRASTRUCTURE_ACTIVE
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRASTRUCTURE_ACTIVE
                name: che
          - name: CHE_INFRA_KUBERNETES_BOOTSTRAPPER_BINARY__URL
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_BOOTSTRAPPER_BINARY__URL
                name: che
          - name: CHE_INFRA_KUBERNETES_MACHINE__START__TIMEOUT__MIN
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_MACHINE__START__TIMEOUT__MIN
                name: che
          - name: CHE_INFRA_KUBERNETES_MASTER__URL
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_MASTER__URL
                name: che
          - name: CHE_INFRA_KUBERNETES_OAUTH__TOKEN
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_OAUTH__TOKEN
                name: che
          - name: CHE_INFRA_KUBERNETES_PVC_STRATEGY
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_PVC_STRATEGY
                name: che
          - name: CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS
                name: che
          - name: CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER
                name: che
          - name: CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP
                name: che
          - name: JAVA_OPTS
            valueFrom:
              configMapKeyRef:
                key: JAVA_OPTS
                name: che
          - name: CHE_INFRA_KUBERNETES_PASSWORD
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_PASSWORD
                name: che
          - name: CHE_INFRA_KUBERNETES_TRUST__CERTS
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_TRUST__CERTS
                name: che
          - name: CHE_INFRA_KUBERNETES_USERNAME
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_USERNAME
                name: che
          - name: CHE_INFRA_KUBERNETES_NAMESPACE
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_NAMESPACE
                name: che
          - name: CHE_LOCAL_CONF_DIR
            valueFrom:
              configMapKeyRef:
                key: CHE_LOCAL_CONF_DIR
                name: che
          - name: CHE_LOGS_DIR
            valueFrom:
              configMapKeyRef:
                key: CHE_LOGS_DIR
                name: che
          - name: CHE_LOG_LEVEL
            valueFrom:
              configMapKeyRef:
                key: CHE_LOG_LEVEL
                name: che
          - name: CHE_MULTIUSER
            valueFrom:
              configMapKeyRef:
                key: CHE_MULTIUSER
                name: che
          - name: CHE_OAUTH_GITHUB_CLIENTID
            valueFrom:
              configMapKeyRef:
                key: CHE_OAUTH_GITHUB_CLIENTID
                name: che
          - name: CHE_OAUTH_GITHUB_CLIENTSECRET
            valueFrom:
              configMapKeyRef:
                key: CHE_OAUTH_GITHUB_CLIENTSECRET
                name: che
          - name: CHE_PREDEFINED_STACKS_RELOAD__ON__START
            valueFrom:
              configMapKeyRef:
                key: CHE_PREDEFINED_STACKS_RELOAD__ON__START
                name: che
          - name: CHE_WORKSPACE_AUTO_START
            valueFrom:
              configMapKeyRef:
                key: CHE_WORKSPACE_AUTO_START
                name: che
          - name: CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON
                name: che
          - name: CHE_INFRA_KUBERNETES_PVC_ENABLED
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_PVC_ENABLED
                name: che
          - name: CHE_INFRA_KUBERNETES_PVC_NAME
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_PVC_NAME
                name: che
          - name: CHE_INFRA_KUBERNETES_PVC_QUANTITY
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_PVC_QUANTITY
                name: che
          - name: CHE_INFRA_KUBERNETES_PVC_ACCESS__MODE
            valueFrom:
              configMapKeyRef:
                key: CHE_INFRA_KUBERNETES_PVC_ACCESS__MODE
                name: che
          image: eclipse/che-server:6.1.0
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /api/system/state
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 120
            timeoutSeconds: 10
          name: che
          ports:
          - containerPort: 8080
            name: http
          - containerPort: 8000
            name: http-debug
          readinessProbe:
            httpGet:
              path: /api/system/state
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 15
            timeoutSeconds: 60
          resources:
            limits:
              memory: 600Mi
            requests:
              memory: 256Mi
          volumeMounts:
          - mountPath: /data
            name: che-data-volume
        serviceAccountName: che
        imagePullSecrets:
        - name: regsecret
    volumeClaimTemplates:
    - metadata:
        name: che-data-volume
      spec:
        storageClassName: manual
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi
        # volumes:
        # - name: che-data-volume
        #   persistentVolumeClaim:
        #     claimName: che-data-volume

OS and version:

Che 6.1

Diagnostics:

Issue Analytics

State:
Created 6 years ago
Comments:8

Top GitHub Comments

1reaction

EriksonMurrugarracommented, Mar 19, 2018

Hi @eivantsov. I work with Cesar on the same team and I applied the changes you mentioned above:

Fist, I have changed from version 6.1.0 to 6.2.0. Secondly, I have ssh into the workspace by running:

kubectl exec -it workspacenusr4pyckhe0z5cf.dockerimage bash -n che

Once in the pod there are two folders /tmp and /projects. Into the /projects folder there is a /tmp/bootstraper/config.json file and into the /workspace_logs/ there is a file /workspace_logs/bootstrapper/bootstrapper.log with the following content: sh: 1: /tmp/bootstrapper/bootstrapper: not found

and we see in the user interface the following message: Could not start workspace wksp-om91. Reason: Waiting for Kubernetes environment ‘default’ of the workspace’workspacel717oeuqk5ybpkj6’ reached timeout

0reactions

ghostcommented, Apr 24, 2018

Closing due to inactivity. Please try the latest release and reopen with updated details

Top Results From Across the Web

workspace creation failure question · Issue #10371 · eclipse/che

I deployed single user eclipse che with kubernetes with ceph storage. I can deploy workspace but it's failing during setup procedure.

Chapter 2. Known issues Red Hat CodeReady Workspaces 2.3

Creating a namespace fails when a login name with unsupported characters is ... CodeReady Workspaces deployed without TLS support causes some features to ......

Troubleshooting deployment of Wazi for Dev Spaces - IBM

You can troubleshoot workspace creation by looking at workspace pod logs. ... Failed to pull image "docker.io/ibmcom/wazi-code-plugin": rpc error: code ...

Common CML Errors and Solutions | CDP Private Cloud

ML workspace provisioning fails because CDP could not get access to all the AWS resources needed to deploy a CML workspace. This is...

Troubleshooting | Google Kubernetes Engine (GKE)

GKE returns an error if there are issues with a workload's Pods. You can check the status of a Pod using the kubectl...