MountVolume.SetUp failed for all volumes

Upon inspecting any pod, I get an error like the following

MountVolume.SetUp failed for volume "kube-dns-token-5nbrm" : couldn't propagate object cache: timed out waiting for the condition

Please run microk8s.inspect and attach the generated tarball to this issue. inspection-report-20191015_164003.tar.gz

We appreciate your feedback. Thank you for using microk8s.

[ghost]

This seems to have resolved itself after a clean remove and reinstall.

EDIT: Nvm. The issue seems to happen when I install rbac first.

Repro: 1) install fresh microk8s with channel=1.15/stable. 2) microk8s.enable rbac, then microk8s.enable storage.

You’ll get the above error. However, if I do microk8s.enable storage first, the host provisioner comes up just fine. Bizarre

[ghost]

1. microk8s.enable storage (storage comes up fine)
2. microk8s.enable rbac
3. microk8s.enable dns

result

microk8s.kubectl describe pod coredns-f7867546d-zzvl5 -n kube-system
Name:                 coredns-f7867546d-zzvl5
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 james-x399-aorus-xtreme/10.0.0.183
Start Time:           Fri, 08 Nov 2019 20:09:32 -0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=f7867546d
Annotations:          scheduler.alpha.kubernetes.io/critical-pod: 
Status:               Pending
IP:                   
Controlled By:        ReplicaSet/coredns-f7867546d
Containers:
  coredns:
    Container ID:  
    Image:         coredns/coredns:1.5.0
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-2wrdp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-2wrdp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-2wrdp
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                              Message
  ----     ------       ----               ----                              -------
  Normal   Scheduled    25s                default-scheduler                 Successfully assigned kube-system/coredns-f7867546d-zzvl5 to james-x399-aorus-xtreme
  Warning  FailedMount  21s (x3 over 24s)  kubelet, james-x399-aorus-xtreme  MountVolume.SetUp failed for volume "coredns-token-2wrdp" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  21s (x3 over 24s)  kubelet, james-x399-aorus-xtreme  MountVolume.SetUp failed for volume "config-volume" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  3s (x5 over 15s)   kubelet, james-x399-aorus-xtreme  MountVolume.SetUp failed for volume "coredns-token-2wrdp" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  3s (x5 over 15s)   kubelet, james-x399-aorus-xtreme  MountVolume.SetUp failed for volume "config-volume" : couldn't propagate object cache: timed out waiting for the condition

1. microk8s.enable storage
2. microk8s.enable dns

result

Everything works fine. Something about microk8s.enable rbac is screwing everything up.