Error out of disk space when install in a LXC container

Hi,

I have an error when I am trying to install microk8s in a LXC container with nesting and privileged security to true. I follow these steps:

1. After access the container, I install it: sudo snap install microk8s --edge --classic
2. Enable dns and dashboard addons: microk8s.enable dns dashboard
3. List all pods: microk8s.kubectl get pods --all-namespaces
4. Access info heapster pod: microk8s.kubectl describe pod heapster-v1.5.2-84f5c8795f-c8hld --namespace kube-system

At that moment I see this info:

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  21m (x6 over 21m)  default-scheduler  0/1 nodes are available: 1 node(s) were not ready.
  Warning  FailedScheduling  21m                default-scheduler  0/1 nodes are available: 1 node(s) were not ready, 1 node(s) were out of disk space.

I have checked my ZFS storage and I have not a problem with my disk space.

Can anyone reproduce it with LXC container? Any ideas?

Thank you very much

[ktsakalozos]

Hi @raguilera82 ,

After some experimentation with LXC and zfs I got microk8s working. Here is what I had to do:

1. Create the following profile:

$ cat microk8s.profile 
name: microk8s
config:
  boot.autostart: "true"
  linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,nf_conntrack_ipv4,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw
    lxc.cgroup.devices.allow=a
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /sys/module/nf_conntrack/parameters/hashsize
    type: disk
  aadisable1:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk
  aadisable2:
    path: /dev/zfs
    source: /dev/zfs
    type: disk
$ lxc profile copy default microk8s
$ cat microk8s.profile | lxc profile edit microk8s

1. Launch a container:

$ lxc launch -p default -p microk8s ubuntu:18.04 

1. Install zfsutils-linux in the container. This package should be part of microk8s so I also opened a new issue.

# apt install zfsutils-linux

1. And finally install microk8s

# snap install microk8s --edge --classic

Most of the work is on the lxc profile so there is not really much to do for microk8s. Here is an ascii cast of the above steps: https://asciinema.org/a/198964

Thanks

[ktsakalozos]

Hi @kim0 ,

I am not entirely sure what you are asking me. I would say it is a little hard to reach the host but not hard enough.

MicroK8s, can get out of the lxc confinement and can access the host machine and this is the purpose of the profile we apply. MicroK8s is expected to run on the host and you are expected to trust it. What you might not trust is the user workload that runs inside Kubernetes and that is constrained by docker containers. You should be aware that MicroK8s is not supposed to be deployed in production or in untrusted environments. Its purpose is mainly to assist in application development locally. If security is of concern you should look into production grade Kubernetes solutions.

Having said that here are a few ways to harden MicroK8s.

• Refine the lxc profile
• Use MicroK8s inside a VM
• Set AppArmor and firewall rules that apply to your usecase
• Test and contribute to this PR https://github.com/ubuntu/microk8s/pull/88

Thanks