microk8s enable kubeflow fails on microk8s 1.21/beta

The kubeflow addon fails to enable on microk8s 1.21/beta. The dex-auth pods are stuck in CrashLoopBackOff status with the following error message in the pod logs:

ubuntu@blanka:~$ microk8s kubectl logs dex-auth-5fcb9bc549-vjmhd -n kubeflow -c dex-auth
time="2021-03-27T18:42:37Z" level=info msg="config using log level: debug"
time="2021-03-27T18:42:37Z" level=info msg="config issuer: http://10.64.140.43.xip.io/dex"
failed to initialize storage: failed to inspect service account token: jwt claim "kubernetes.io/serviceaccount/namespace" not found

I also notice an error with the metadata-api pod:

ubuntu@blanka:~$ microk8s kubectl logs metadata-api-59c98cbdb9-qrwsz -n kubeflow
E0327 18:39:50.014738       1 register.go:68] Ignored unknown category "container" with type "workspace" in "http://github.com/kubeflow/metadata/schema/alpha/containers/workspace.json"
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xe2419b]
[...]

The same command microk8s enable kubeflow works on microk8s channels 1.19 and 1.20, but not on the 1.21/beta channel.

I tried on both a Nvidia DGX A100 as well as on a ProLiant DL380 Gen10 and the result is the same.

ubuntu@blanka:~$ microk8s kubectl get pods -A | grep -v Running NAMESPACE NAME READY STATUS RESTARTS AGE ingress nginx-ingress-microk8s-controller-sm57t 0/1 CrashLoopBackOff 9 21m kubeflow dex-auth-5854d6666f-fc6fr 1/2 CrashLoopBackOff 6 12m kubeflow dex-auth-5fcb9bc549-vjmhd 1/2 CrashLoopBackOff 6 12m kubeflow metadata-api-59c98cbdb9-qrwsz 0/1 CrashLoopBackOff 7 14m

Additional information:

ubuntu@blanka:~$ snap list microk8s
Name      Version         Rev   Tracking   Publisher   Notes
microk8s  v1.21.0-beta.1  2085  1.21/beta  canonical✓  classic

ubuntu@blanka:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

inspection-report-20210327_183701.tar.gz

[DavidSpek]

@mat-ej There is a fix stated here. Also ping @RFMVasconcelos

[davecore82]

The following workaround lets me use Kubeflow and GPU with microk8s:

1. sudo snap install microk8s --channel=1.20 --classic
2. microk8s enable kubeflow
3. sudo snap refresh microk8s --channel=1.21/beta
4. microk8s enable gpu

There’s a few things required to enable gpu. See https://github.com/ubuntu/microk8s/issues/448#issuecomment-805122344 for details.