Cluster join no longer works after enabling Cilium

During the setup process, Cilium addon disables the cni.yaml file for microk8s. This causes cluster join to fail due to CNI autodetection. This code path has plenty of comments about Calico specific operations, so it seems that Cilium needs some work on this front.

Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]: Waiting for access to cluster. [2021-04-11 14:04:43,340] ERROR in app: Exception on /cluster/api/v2.0/join [POST]
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]: Traceback (most recent call last):
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     response = self.full_dispatch_request()
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     rv = self.handle_user_exception(e)
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     reraise(exc_type, exc_value, tb)
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     raise value
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     rv = self.dispatch_request()
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     return self.view_functions[rule.endpoint](**req.view_args)
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/scripts/cluster/agent.py", line 594, in join_node_dqlite
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     try_initialise_cni_autodetect_for_clustering(node_addr, apply_cni=True)
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/scripts/cluster/common/utils.py", line 240, in try_initialise_cni_autodetect_for_clustering
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     if cni_is_patched():
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:   File "/snap/microk8s/2094/scripts/cluster/common/utils.py", line 207, in cni_is_patched
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]:     with open(yaml) as f:
Apr 11 14:04:43 wanted-wolf microk8s.daemon-cluster-agent[1598267]: FileNotFoundError: [Errno 2] No such file or directory: '/var/snap/microk8s/2094/args/cni-network/cni.yaml'

Tested with the microk8s snap package from 1.20/stable channel:

snap-id:      EaXqgt1lyCaxKaQCU349mlodBkDCXRcg
tracking:     1.20/stable
refresh-date: yesterday at 21:33 UTC
channels:
  1.20/stable:      v1.20.5  2021-04-02 (2094) 218MB classic

[ktsakalozos]

Thank you for reporting this @ressu . The cilium addon is not suited for multi-node clusters. However we should not fail in such an ugly way. We should be warning the user and fail gracefully.

[PRNDA]

The cilium addon is not suited for multi-node clusters

But why?