Ubuntu 18.04.02 Broken Since Package Upgrade (coredns "Pod sandbox changed, it will be killed...")

I’m breaking this out as a separate issue from my comments here @ktsakalozos , where I described that all my deployments stopped working on a machine that had been untouched for ~10 days simply because I did an apt-get upgrade and rebooted. No other changes were made.

The following steps reproduce the issue, which I’ve tried on both stable and edge channels. You can find the output of microk8s.inspect here

Each time, I ran just the following commands:

microk8s.reset
snap remove microk8s
sudo snap install microk8s --edge --classic
microk8s.enable dns storage

Between the uninstall & reinstall, I have also tried verifying that /snap/microk8s and /var/snap/microk8s and cat /proc/mounts | grep microk8 are all empty. I have also tried rebooting at that time and double-checking that there are no updates via snap refresh microk8s. If I run microk8s.kubectl get pods --all-namespaces -o wide I see:

  IP           NODE    NOMINATED NODE   READINESS GATES
kube-system   coredns-5874dcd95f-sn2gh                1/1     Running   11         2m49s   10.1.1.144   house   <none>           <none>
kube-system   hostpath-provisioner-6d744c4f7c-pkhqj   1/1     Running   5          2m42s   10.1.1.143   house   <none>           <none>

Describing one of the pods, I see:

Name:               coredns-5874dcd95f-sn2gh
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               house/192.168.1.100
Start Time:         Thu, 13 Jun 2019 20:59:10 -0700
Labels:             k8s-app=kube-dns
                    pod-template-hash=5874dcd95f
Annotations:        scheduler.alpha.kubernetes.io/critical-pod: 
Status:             Running
IP:                 10.1.1.149
Controlled By:      ReplicaSet/coredns-5874dcd95f
Containers:
  coredns:
    Container ID:  containerd://ec8410c66229805b25abc5560d3d30d02f39e63dd64ce5e01249e46582a52c60
    Image:         coredns/coredns:1.5.0
    Image ID:      sha256:7987f0908caf347ac98bc23ef590eaf659dcdca661d57fdada563ccfe7edb043
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Thu, 13 Jun 2019 21:02:30 -0700
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Thu, 13 Jun 2019 21:02:18 -0700
      Finished:     Thu, 13 Jun 2019 21:02:28 -0700
    Ready:          False
    Restart Count:  13
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-t5bgj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-t5bgj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-t5bgj
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               3m22s                  default-scheduler  Successfully assigned kube-system/coredns-5874dcd95f-sn2gh to house
  Normal   Pulling                 3m19s                  kubelet, house     Pulling image "coredns/coredns:1.5.0"
  Normal   Pulled                  3m15s                  kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 3m14s                  kubelet, house     Created container coredns
  Normal   Started                 3m14s                  kubelet, house     Started container coredns
  Normal   SandboxChanged          3m4s                   kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  3m3s                   kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 3m3s                   kubelet, house     Created container coredns
  Normal   Started                 3m3s                   kubelet, house     Started container coredns
  Normal   SandboxChanged          2m58s (x2 over 2m58s)  kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  2m57s                  kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 2m57s                  kubelet, house     Created container coredns
  Normal   Started                 2m57s                  kubelet, house     Started container coredns
  Normal   SandboxChanged          2m39s (x2 over 2m40s)  kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  2m39s                  kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 2m39s                  kubelet, house     Created container coredns
  Warning  Failed                  2m39s                  kubelet, house     Error: transport is closing
  Normal   SandboxChanged          2m33s (x2 over 2m34s)  kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  2m33s                  kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 2m32s                  kubelet, house     Created container coredns
  Normal   Started                 2m32s                  kubelet, house     Started container coredns
  Normal   SandboxChanged          2m13s (x2 over 2m13s)  kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  2m12s                  kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Warning  Failed                  2m11s                  kubelet, house     Error: transport is closing
  Normal   SandboxChanged          2m5s (x2 over 2m6s)    kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  2m5s                   kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 2m4s                   kubelet, house     Created container coredns
  Normal   Started                 2m4s                   kubelet, house     Started container coredns
  Normal   SandboxChanged          107s (x2 over 107s)    kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  106s                   kubelet, house     Failed create pod sandbox: rpc error: code = Unavailable desc = transport is closing
  Normal   SandboxChanged          101s                   kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  101s                   kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Started                 100s                   kubelet, house     Started container coredns
  Normal   Created                 100s                   kubelet, house     Created container coredns
  Normal   SandboxChanged          88s (x2 over 89s)      kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  88s                    kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 88s                    kubelet, house     Created container coredns
  Normal   Started                 87s                    kubelet, house     Started container coredns
  Normal   SandboxChanged          70s (x2 over 71s)      kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  69s                    kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 69s                    kubelet, house     Created container coredns
  Warning  Failed                  69s                    kubelet, house     Error: transport is closing
  Normal   SandboxChanged          64s (x2 over 64s)      kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  63s                    kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 63s                    kubelet, house     Created container coredns
  Normal   Started                 63s                    kubelet, house     Started container coredns
  Normal   SandboxChanged          46s (x2 over 46s)      kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  45s                    kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 45s                    kubelet, house     Created container coredns
  Warning  Failed                  45s                    kubelet, house     Error: transport is closing
  Normal   SandboxChanged          39s (x2 over 40s)      kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  38s                    kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 38s                    kubelet, house     Created container coredns
  Normal   Started                 37s                    kubelet, house     Started container coredns
  Normal   SandboxChanged          21s (x2 over 22s)      kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  20s                    kubelet, house     Failed create pod sandbox: rpc error: code = Unavailable desc = transport is closing
  Normal   SandboxChanged          15s                    kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  15s                    kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 15s                    kubelet, house     Created container coredns
  Normal   Started                 14s                    kubelet, house     Started container coredns
  Normal   SandboxChanged          3s (x2 over 3s)        kubelet, house     Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  2s                     kubelet, house     Container image "coredns/coredns:1.5.0" already present on machine
  Normal   Created                 2s                     kubelet, house     Created container coredns
  Normal   Started                 2s                     kubelet, house     Started container coredns

[ktsakalozos]

We have a service that detects changes in the network so as to reconfigure the services accordingly. Can you try to stop that service and see if the service restarts stop? Try:

sudo systemctl stop snap.microk8s.daemon-apiserver-kicker.service

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.