Unable to start a workspace on fresh minikube/che install [v7.8.0]

Describe the bug

I’ve created a local minikube cluster following the documentation, and deployed che to it using chectl. When creating and opening a workspace from any stack, the following log appears until it fails in a timeout.

Successfully assigned che/workspacevljc0zfnpzuh4u76.che-plugin-broker to minikube
Pulling image "quay.io/eclipse/che-plugin-metadata-broker:v3.1.0"
Successfully pulled image "quay.io/eclipse/che-plugin-metadata-broker:v3.1.0"
Created container che-plugin-metadata-broker-v3-1-0
Started container che-plugin-metadata-broker-v3-1-0
Error: Failed to run the workspace: "Plugins installation process timed out"

I’ve put some further investigation to the “additional context” section.

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

1. Spin up minikube minikube start --vm-driver=kvm2 --memory=6144 --cpus=6 --disk-size=32gb --dns-domain='kube.lab'
2. Start che chectl server:start --platform minikube --domain "kube.lab"
3. Put che hostnames in /etc/hosts on workstation
4. Visit the che server and start a workspace

Expected behavior

The workspace should start and open correctly.

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (minikube version: v1.7.2 , kubectl version: v1.17.2)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl (server:start --platform minikube --domain "kube.lab")
• che-operator
• minishift-addon
• I don’t know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Additional context

I’ve tried this on recently set up latest debian buster and latest manjaro linux (amd64). Helm, chectl, minikube and system apckages are all up to date. Host firewall is turned off for testing. KVM is properly running and configured. Changing the domain, or leaving it on default has no effect on the current behavior.

The kubernetes dashboard shows all PODs/services to be running and healthy, but when starting a workspace in che, the workspace POD fails with following log.

2020/02/12 11:50:31 Broker configuration
2020/02/12 11:50:31   Push endpoint: ws://che-che.kube.lab/api/websocket
2020/02/12 11:50:31   Auth enabled: false
2020/02/12 11:50:31   Runtime ID:
2020/02/12 11:50:31     Workspace: workspacevljc0zfnpzuh4u76
2020/02/12 11:50:31     Environment: default
2020/02/12 11:50:31     OwnerId: che
2020/02/12 11:50:31 Couldn't connect to endpoint 'ws://che-che.kube.lab/api/websocket', due to error 'dial tcp: lookup che-che.kube.lab on 10.96.0.10:53: no such host'

The kube-dns service is running, and mapped to two coredns PODs. I’ve created a dnsutils POD in namespace che to investigate possible DNS issues. From there neither nslookup, nor dig can resolve the che server hostname no matter what domain to query for.

# nslookup che-che.kube.lab
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find che-che.kube.lab: NXDOMAIN

I don’t know whether this is just a configuration issue, or a fault in che, or minikube. As I’m new to kubernetes/che, I don’t know where to go from here. Searching the web results in many issues related to older versions of minikube/che and their fixes do not work at this point. Also the che and minikube documentations provide absolutely no hint on this, or even exact configuration steps. So I’m considering the described steps there as a correct how-to (at least for first steps), but they are simply not working (no rant, just a perception). It would be great if we can find a solution on this, and maybe extend the documentation by all necessary steps.

[tolusha]

@Jarthianur Thank you, we will do.

[tolusha]

The following scenario works without setting up external dns server:

1. minikube start --vm-driver=virtualbox --memory=6144 --cpus=6 --disk-size=32gb --dns-domain=‘kube.lab’
2. kubectl -n kube-system edit configmap coredns and add cluster.local to the configuration

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes kube.lab cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap

1. chectl server:start --platform minikube --domain $(minikube ip).nip.io
2. create a workspace

https://github.com/eclipse/che/issues/14404