Provide support for ZFS

I would like full support for ZFS in microk8s. If microk8s sees that the root fs is ZFS it should use settings that work on ZFS.

background

With the newest release (492) microk8s switched from docker to containerd. Because I use ZFS on all my disks this did not go well.

I had previously altered args/docker-daemon.json with the directions here: https://docs.docker.com/storage/storagedriver/zfs-driver/ (adding "storage-driver": "zfs")

With the snap autoupgrade this left me with a broken k8s, where none of the pods would start.

After some investigation I attempted to switch containerd to use zfs:

I updated containerd-template.toml changing snapshotter = "overlayfs" to snapshotter = "zfs" and restarted. Then it complained about a missing metadata.db. Creating a new ZFS dataset and mounting it where it expected it made that error go away, but no pods start. There were errors logged but I don’t recall what they were.

I apologize for missing information in this issue. The system in question is offline due to a hardware problem, but when It comes back up I will edit in more details.

[myers]

see note at bottom!

so you want to use microk8s on ZFS?

1. Stop microk8s. microk8s.stop
2. Remove old state of containerd. sudo rm -rf /var/snap/microk8s/common/var/lib/containerd.
3. Configure containerd to use ZFS. Edit /var/snap/microk8s/current/args/containerd-template.toml replacing "snapshotter = “overlayfs” to snapshotter = “zfs”.
4. Create new zfs dataset for containerd to use. zfs create -o mountpoint=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.zfs $POOL/containerd
5. Start microk8s. microk8s.start

Me: Two years later

Having used the zfs snapshotter for the past 2 years I’d like to throw my $0.02 in. DON’T. The current verison of ZFS tools (like the zfs cli) are not ready for how the zfs snapshotter uses datasets. Docker images are made up of layers, 10’s up to 100’s of layers per image. Each of the these layers get created as a dataset in zfs. On my system there are over 20,000 of these datasets. zfs list takes over a minute to run. update-grub takes 6 minutes to run because it does zfs list once for every pool in your system (probably something they can improve).

I suspect the snapshotter maybe loosing track of these datasets, as I think 20,000 seems a bit high.

I am now running microk8s on a zvol. That’s a part of a zfs pool that you can format as any filesystem. The big downside is that it’s a fixed size.

how to run microk8s on zfs using a zvol

The pool we want to store the microk8s image files and container fs is rpool. We want to make a 900GB volume. This will make you download all the images for your pods again.

As root do:

microk8s.stop
mv /var/snap/microk8s/common/var/lib/containerd /var/snap/microk8s/common/var/lib/containerd.backup
mkdir /var/snap/microk8s/common/var/lib/containerd
zfs create -V 900gb rpool/containerd
mkfs.ext4 /dev/zvol/rpool/containerd
echo "/dev/zvol/rpool/containerd /var/snap/microk8s/common/var/lib/containerd ext4 errors=remount-ro 0 2" >> /etc/fstab
mount /var/snap/microk8s/common/var/lib/containerd
microk8s.start

[qlyoung]

@ktsakalozos this is still being hit on a stock install, 9 months later. Is it possible to at least get a doc change? Honestly, snaps don’t work on nonstandard homedirs, snaps don’t work on nfs, microk8s doesn’t work on zfs…all these would be great to document in a “gotchas” section in the user guide.