kubelet cannot execute runc

Please run microk8s inspect and attach the generated tarball to this issue.

We appreciate your feedback. Thank you for using microk8s inspection-report-20211012_103831.tar.gz .

$ kubectl describe pod kube-registry-699c4fd457-skkvd
...
Events:
  Type     Reason         Age                   From     Message
  ----     ------         ----                  ----     -------
  Warning  FailedKillPod  17m                   kubelet  error killing pod: [failed to "KillContainer" for "registry" with KillContainerError: "rpc error: code = Unknown desc = failed to stop container \"d30b2fa7037f0abeb639fc89afffe1319ed57642bbc8431b99f851e7f792496c\": unknown error after kill: exec: \"runc\": executable file not found in $PATH: <nil>: unknown", failed to "KillPodSandbox" for "ff5457f3-73a4-4c8f-a9cc-569214da724a" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container \"d30b2fa7037f0abeb639fc89afffe1319ed57642bbc8431b99f851e7f792496c\": failed to kill container \"d30b2fa7037f0abeb639fc89afffe1319ed57642bbc8431b99f851e7f792496c\": unknown error after kill: exec: \"runc\": executable file not found in $PATH: <nil>: unknown"]
  Normal   Killing        9m31s (x13 over 17m)  kubelet  Stopping container registry
  Warning  FailedKillPod  2m34s (x21 over 16m)  kubelet  error killing pod: [failed to "KillContainer" for "registry" with KillContainerError: "rpc error: code = Unknown desc = failed to kill container \"d30b2fa7037f0abeb639fc89afffe1319ed57642bbc8431b99f851e7f792496c\": unknown error after kill: exec: \"runc\": executable file not found in $PATH: <nil>: unknown", failed to "KillPodSandbox" for "ff5457f3-73a4-4c8f-a9cc-569214da724a" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container \"d30b2fa7037f0abeb639fc89afffe1319ed57642bbc8431b99f851e7f792496c\": failed to kill container \"d30b2fa7037f0abeb639fc89afffe1319ed57642bbc8431b99f851e7f792496c\": unknown error after kill: exec: \"runc\": executable file not found in $PATH: <nil>: unknown"]

– Diagnostics results: microk8s has KillMode=process in /etc/systemd/system/snap.microk8s.daemon-containerd.service. On snap update containerd is restarted, but child containers are left running. This is done to allow updates without disturbing cluster payload. When a container survives 2 snap updates, the snap version that launched the container is deleted. Since each version is an independent package, a running container has no management tools after the second snap update.

The solution needs to place a symlink to runc in /snap/bin/and ensure that it is called from there. The real solution is to provide a deb package for microk8s and install runc to /usr/bin/. For now I use the simplest workaround which is to disable automatic snap updates by setting a fake proxy:

snap set system proxy.http="http://127.0.0.1:1111"
snap set system proxy.https="http://127.0.0.1:1111"

When I am ready to update:

snap unset system proxy.http
snap unset system proxy.https
snap refresh
snap set system proxy.http="http://127.0.0.1:1111"
snap set system proxy.https="http://127.0.0.1:1111"

[ktsakalozos]

@yanovich, all, thank you for the awesome debugging work you did in this issue. Let me try to re-iterate what the problem is. When containerd spawns a new container it does so through a containerd-shim. The containerd-shim is the one calling runc when needed therefore it needs to always find it in its PATH. containerd-shims inherit their PATH from containerd. Contrary to containerd the shims persist through snap refreshes. In a refresh containerd gets updated with the new location of runc but the shims do not. The fix (linked to this issue above) is to have the /snap/microk8s/current/bin/runc in the containerd’s PATH so it gets inherited by the shims.

For the workaround while we this patch makes its way through the release process is to put a runc in the PATH of the containerd-shims. Here are a couple of ways:

• apt install runc on the node,
• create a symbolic lint towards the runc of the snap: sudo ln -s /snap/microk8s/current/bin/runc /snap/bin/runc . Other locations should also work, eg sudo ln -s /snap/microk8s/current/bin/runc /usr/bin/runc.

Another workaround is to reboot the node.

Apologies for the inconvenience we may have caused. We are working to fix this.

[yanovich]

microk8s has KillMode=process in /etc/systemd/system/snap.microk8s.daemon-containerd.service. On snap update containerd is restarted, but child containers are left running. This is done to allow updates without disturbing cluster payload. When a container survives 2 snap updates, the snap version that launched the container is deleted. Since each version is an independent package, a running container has no management tools after the second snap update.

The solution needs to place a symlink to runc in /snap/bin/and ensure that it is called from there. The real solution is to provide a deb package for microk8s and install runc to /usr/bin/. For now I use the simplest workaround which is to disable automatic snap updates by setting a fake proxy:

snap set system proxy.http="http://127.0.0.1:1111"
snap set system proxy.https="http://127.0.0.1:1111"

When I am ready to update:

snap unset system proxy.http
snap unset system proxy.https
snap refresh
snap set system proxy.http="http://127.0.0.1:1111"
snap set system proxy.https="http://127.0.0.1:1111"