microk8s on Ubuntu 20.04.1 LTS - Offline Install. Nodes "NotReady"

While adding the tarball is possible I suppose, it sort of defeats the purpose of having an air gapped system.

I am in the process of testing out microk8s, and have installed 3 fresh Ubuntu 20.04.1 LTS VMs running on Nutanix hardware, they each have 4 vCPU and 8 GB RAM with a 60GB disk for OS and a 100GB disk that I have set aside for Ceph (so is not yet in use).

Have run an offline install using snap ack/snap install - and everything looks fine, but it obviously isn’t.

 $ df -hT

Filesystem                             Type      Size  Used Avail Use% Mounted on
udev                                   devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs                                  tmpfs     791M  1.2M  790M   1% /run
/dev/mapper/vg_ubuntu-lv_root          xfs        26G  7.3G   19G  28% /
tmpfs                                  tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs                                  tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                                  tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0                             squashfs   99M   99M     0 100% /snap/core/10823
/dev/loop1                             squashfs   56M   56M     0 100% /snap/core18/1988
/dev/loop3                             squashfs   72M   72M     0 100% /snap/lxd/19389
/dev/loop2                             squashfs   32M   32M     0 100% /snap/snapd/11036
/dev/mapper/vg_ubuntu-lv_var_tmp       xfs       2.0G   35M  2.0G   2% /var/tmp
/dev/mapper/vg_ubuntu-lv_tmp           xfs      1014M   34M  981M   4% /tmp
/dev/mapper/vg_ubuntu-lv_home          xfs        20G  629M   20G   4% /home
/dev/sda2                              xfs      1014M  215M  800M  22% /boot
/dev/mapper/vg_ubuntu-lv_var_log       xfs       5.0G  423M  4.6G   9% /var/log
/dev/mapper/vg_ubuntu-lv_var_log_audit xfs       5.0G   40M  5.0G   1% /var/log/audit
tmpfs                                  tmpfs     791M     0  791M   0% /run/user/1000
/dev/loop4                             squashfs  208M  208M     0 100% /snap/microk8s/1910

 $ snap list
Name      Version      Rev    Tracking  Publisher   Notes
core      16-2.48.2.1  10823  -         canonical✓  core
core18    20210128     1988   -         canonical✓  base
lxd       4.11         19389  -         canonical✓  -
microk8s  v1.20.1      1910   -         canonical✓  classic
snapd     2.48.2.1     11036  -         canonical✓  snapd

$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.

$ microk8s kubectl describe nodes

Name:               my-microk8s-server1
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=my-microk8s-server1
                    kubernetes.io/os=linux
                    microk8s.io/cluster=true
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 16 Feb 2021 13:55:51 +0100
Taints:             node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  my-microk8s-server1
  AcquireTime:     <unset>
  RenewTime:       Tue, 16 Feb 2021 13:56:51 +0100
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 16 Feb 2021 13:56:01 +0100   Tue, 16 Feb 2021 13:55:51 +0100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 16 Feb 2021 13:56:01 +0100   Tue, 16 Feb 2021 13:55:51 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 16 Feb 2021 13:56:01 +0100   Tue, 16 Feb 2021 13:55:51 +0100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Tue, 16 Feb 2021 13:56:01 +0100   Tue, 16 Feb 2021 13:55:51 +0100   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Addresses:
  InternalIP:  192.168.0.50
  Hostname:    my-microk8s-server1
Capacity:
  cpu:                4
  ephemeral-storage:  27245572Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8099644Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  26196996Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             7997244Ki
  pods:               110
System Info:
  Machine ID:                 7ade511b291b45d8a2c68362c619ab63
  System UUID:                7ade511b-291b-45d8-a2c6-8362c619ab63
  Boot ID:                    51a48642-92ad-4db9-bbe2-9bc856a9d70b
  Kernel Version:             5.4.0-53-generic
  OS Image:                   Ubuntu 20.04.1 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.3.7
  Kubelet Version:            v1.20.1-34+e7db93d188d0d1
  Kube-Proxy Version:         v1.20.1-34+e7db93d188d0d1
Non-terminated Pods:          (1 in total)
  Namespace                   Name                 CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                 ------------  ----------  ---------------  -------------  ---
  kube-system                 calico-node-p2gq7    250m (6%)     0 (0%)      0 (0%)           0 (0%)         50s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                250m (6%)  0 (0%)
  memory             0 (0%)     0 (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:
  Type     Reason                   Age   From        Message
  ----     ------                   ----  ----        -------
  Normal   Starting                 65s   kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      65s   kubelet     invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  65s   kubelet     Node my-microk8s-server status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    65s   kubelet     Node my-microk8s-server status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     65s   kubelet     Node my-microk8s-server status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  65s   kubelet     Updated Node Allocatable limit across pods
  Normal   Starting                 62s   kube-proxy  Starting kube-proxy.

So the problem seems to be related to Warning InvalidDiskCapacity 65s kubelet invalid capacity 0 on image filesystem, which is something I’ve read about other people experiencing, but then usually when running on zfs.

I tried attaching a “scrubbed” tar.gz file where all my node names and addresses have been changed, but it didn’t work for some reason… I probably messed it up somehow.

Does anyone have any bright ideas about where to start with this one?

Any help is appreciated.

-Marius

[rere8020]

I am having the same problem. I am using Ubuntu 20.04 and I have one server on the internet and another offline. I installed core_10859.assert core_10859.snap microk8s_2035.assert microk8s_2035.snap on the offline one from my online server. My online server is working perfectly. My offline has the same issue as above.

My questions are as follows for the offline system:

1. What will happen if I want to enable add-ons? Won’t it try to go online and pull images?
2. How do I configure containerd to pull from a local registry if the solution is to have a local registry?
3. Do I use a local docker registry container to hold the above images? If not what is the solution?
4. Will it be a problem installing docker and using containerd?

My apologies if I sound foolish asking. I want to make sure I understand clearly and this is the first time looking into microk8s. Thanks in advance!

[balchua]

Hi based on the output you showed, the network is not available.

Ready            False   Tue, 16 Feb 2021 13:56:01 +0100   Tue, 16 Feb 2021 13:55:51 +0100   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Probably calico images are not getting pulled. You can check if calico containers are started properly.

Since you are installing it in an air-gapped environment, you will need to grab the calico images. This file can point you to the actual image used.
You also need to have the pause container image available in your environment. https://github.com/ubuntu/microk8s/blob/11eba423adf5a57f6bb318dbf7b02db60d515cb2/microk8s-resources/default-args/containerd-template.toml#L30