Microk8s cluster nodes stuck in NotReady state on ARM64

Hi all,

After refreshing my snaps to fix the SSL Issue (https://github.com/ubuntu/microk8s/issues/720), my microk8s nodes seem to be acting better but still have some issues.

First of all when I run microk8s.status on any node, it reports that microk8s is not running, despite running microk8s.start and this is the case irrelevant of if a microk8s node is part of a cluster or just a standalone node.

When I ran the microk8s.inspect command on each node, it did not report any errors.

I decided to join all of the machines into a single cluster anyway, but I noticed that the nodes are stuck in a NotReady status:

ubuntu@battleship:~$ microk8s.kubectl get nodes
NAME                     STATUS     ROLES    AGE     VERSION
battlecruiser.microk8s   NotReady   <none>   3h43m   v1.16.1
battleship.microk8s      NotReady   <none>   41h     v1.16.1
submarine.microk8s       NotReady   <none>   3h54m   v1.16.1
ubuntu                   NotReady   <none>   41h     v1.16.0 

I tried to deploy some containers:

microk8s.kubectl run hello-world --replicas=5 --labels="run=load-balancer-example" --image=gcr.io/google-samples/node-hello:1.0 --port=8080

However they would not start:

NAMESPACE   NAME                           READY   STATUS    RESTARTS   AGE
default     hello-world-77b74d7cc8-7h4nr   0/1     Pending   0          8m16s
default     hello-world-77b74d7cc8-9kv46   0/1     Pending   0          8m16s
default     hello-world-77b74d7cc8-ct7zk   0/1     Pending   0          8m16s
default     hello-world-77b74d7cc8-dj7wj   0/1     Pending   0          8m16s
default     hello-world-77b74d7cc8-lt4lc   0/1     Pending   0          8m16s

A quick look deeper:

 Warning  FailedScheduling  <unknown>  default-scheduler  0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.

My Belief is that one of the services is still not happy, or there is an issue with the Ubuntu image (perhaps cgroups related). I will try some further testing/analysis later.

My particular test platform is Raspberry Pi 4 Model B 4GB (x3).

Inspection reports attached: battleship-inspection-report-20191015_144755.tar.gz submarine-inspection-report-20191015_144633.tar.gz battlecruiser-inspection-report-20191015_144639.tar.gz

Cheers,

• Calvin

[ktsakalozos]

I forgot to reply to this issue sorry.

K8s was complaining that the cgroup memory was not enabled. Looking at /proc/cgroups was showing memeory disabled. Changes in config.txt are not enough, what actually worked is editing /boot/firmware/nobtcmd.txt, appending cgroup_enable=memory cgroup_memory=1. and rebooting.

[m0khalifa]

Same issue here, tried the appending cgroup_enable=memory cgroup_memory=1 to /boot/firmware/nobtcmd.txt did not help on ubuntu 20.04 and a rb pi4. Appending it to /boot/firmware/cmdline.txt itself did the trick like @korhojoa mentioned.

For anyone who stumbles on this from google like I did and is stuck.

sudo nano /boot/firmware/cmdline.txt at the end of the file add cgroup_enable=memory cgroup_memory=1 reboot and repeat for each of your nodes if you’re setting up a cluster.