Apiserver stopped working (no changes made)

I have two clusters that have both stopped working in the same way. kubectl can’t connect to them and the apiserver is not running. One of the clusters is a single-node (inspection report comes from that one), the other has three nodes. No changes have been made to the machines where they are running.

I tried upgrading the single-node cluster from v1.21 to 1.22 without any changes. It’s not the problem from #2486, both info.yaml and cluster.yaml have the expected contents…

I don’t know why apiserver isn’t included in the inspect… This is what is says (after upgrading to 1.22, that’s why it’s 2585 here instead of 2546 in the inspect tarball that was created before the upgrade).

● snap.microk8s.daemon-apiserver.service - Service for snap application microk8s.daemon-apiserver
   Loaded: loaded (/etc/systemd/system/snap.microk8s.daemon-apiserver.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2021-10-18 11:26:40 UTC; 2 days ago
  Process: 33264 ExecStart=/usr/bin/snap run microk8s.daemon-apiserver (code=exited, status=0/SUCCESS)
 Main PID: 33264 (code=exited, status=0/SUCCESS)

Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: ++ /snap/microk8s/2585/bin/uname -m
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + ARCH=x86_64
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + export LD_LIBRARY_PATH=:/snap/microk8s/2585/lib:/snap/microk8s/2585/usr/lib:/snap/microk8s/2585/lib/x86_64-linux-gnu:/snap/mic
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + LD_LIBRARY_PATH=:/snap/microk8s/2585/lib:/snap/microk8s/2585/usr/lib:/snap/microk8s/2585/lib/x86_64-linux-gnu:/snap/microk8s/2
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/2585/lib:/snap/microk
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/2585/lib:/snap/microk8s/2585
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + '[' -e /var/snap/microk8s/2585/var/lock/lite.lock ']'
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + echo 'Will not run along with kubelite'
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: Will not run along with kubelite
Oct 18 11:26:40 pm-cluster microk8s.daemon-apiserver[33264]: + exit 0

inspection-report-20211018_080210.tar.gz

[balchua]

The fix on the dqlite to have a more robust write to cluster.yaml is being merged to the different versions. That should fix the single node issue.

[balchua]

@osirisguitar thanks for reporting. Kubelite wraps most if not all kube control plane components, thats why the log you see on the apiserver says it is not starting.

Looking at the logs, it looks like dqlite isn’t starting or it is unable to find the leader. @ktsakalozos @mathieubordere any thoughts? Thanks.