cilium on microk8s: core-dns, hostpath-provisioner and hubble fails.

Enabling cilium on microk8s results into cluster not working.

• coredns stops working
• hospath pod stops working looking into pod describe state shows warning (not sure if related): failed to find plugin “cilium-cni” in path [/var/snap/microk8s/2551/opt/cni/bin]

ENV:

Distributor ID:	Ubuntu
Description:	Ubuntu 21.10
Release:	21.10
Codename:	impish
microk8s 1.22/stable (Single node physical server)

inspection-report-20211018_140644.tar.gz

Plugins

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    cilium               # SDN, fast with full network policy
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    helm3                # Helm 3 - Kubernetes package manager
    metallb              # Loadbalancer for your Kubernetes cluster
    storage              # Storage class; allocates storage from host directory

Enabling cilium plugin causes

• dns pod fails (No policy applied)
• hostPath provision fails (storage)

.:53
[INFO] plugin/reload: Running configuration MD5 = be0f52d3c13480652e0c73672f2fa263
CoreDNS-1.8.0
linux/amd64, go1.15.3, 054c9ae
[INFO] 127.0.0.1:38711 - 2364 "HINFO IN 8559940237512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 6.001203496s
[ERROR] plugin/errors: 2 8559940237512818014.7305508818995797580. HINFO: read udp 10.0.0.146:50304->8.8.8.8:53: i/o timeout
[INFO] 127.0.0.1:40286 - 35177 "HINFO IN 8559940237512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 6.000765939s
[ERROR] plugin/errors: 2 8559940237512818014.7305508818995797580. HINFO: read udp 10.0.0.146:33405->8.8.4.4:53: i/o timeout
[INFO] 127.0.0.1:58037 - 43993 "HINFO IN 8559940237512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 4.000622412s
[ERROR] plugin/errors: 2 8559940237512818014.7305508818995797580. HINFO: read udp 10.0.0.146:60612->8.8.4.4:53: i/o timeout
I1018 12:06:34.490262       1 trace.go:205] Trace[774965466]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (18-Oct-2021 12:06:04.489) (total time: 30000ms):
Trace[774965466]: [30.000588379s] [30.000588379s] END
E1018 12:06:34.490290       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://10.152.183.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.152.183.1:443: i/o timeout
I1018 12:06:34.490319       1 trace.go:205] Trace[436340495]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (18-Oct-2021 12:06:04.489) (total time: 30000ms):

Looking at pod describe:

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Warning  FailedScheduling        66m                    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Warning  FailedScheduling        65m                    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled               58m                    default-scheduler  Successfully assigned kube-system/coredns-7f7f9dc75b-fq7wx to antaresinc-cluster
  Warning  FailedCreatePodSandBox  58m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf739b3ee917271f23fd6187b9e9f5f6417ecca0a942aba36acbf2bc25ee0b80": failed to find plugin "cilium-cni" in path [/var/snap/microk8s/2551/opt/cni/bin]

Cilium Hubble

Warning  FailedCreatePodSandBox  34m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d54e5a6079466384ab662149f9dc6f3c6865a82669b06a8b953d0bcc84817f8f": failed to find plugin "cilium-cni" in path [/var/snap/microk8s/2551/opt/cni/bin]

[usma0118]

I have downgraded Ubuntu 20.04 and cilium seems to be working now.

[joestringer]

I think the next steps would probably need to involve trying something like using an older Ubuntu version to see whether this is a regression in the combination of Ubuntu 21.10 and Cilium 1.10.x. I can tell you that I was able to successfully install microk8s+cilium on Ubuntu 20.04 LTS here with a Linux 5.10.x kernel. If you’re able to try that out, it could help to confirm whether there is a problem with the integration to Ubuntu 21.10 or whether there is another problem in your environment.

In the mean time, I reached out to the cilium.io Slack community #development channel to see whether any others have experience running Cilium on this new version of Ubuntu yet, so hopefully we can find out whether that’s the problem and whether there are any mitigations or fixes there.