Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

After upgrading to 21/stable all my pods fail to create

See original GitHub issue

Hi, I wanted to ask if anyone has tried to upgrade fron 19/stable to 21/stable and experienced the same issue, every single container faces the issue. (i for now downgraded to 20/stable and everything is working)

sofo@k8s-test-m:~$ kubectl get all --all-namespaces
NAMESPACE                  NAME                                                          READY   STATUS    RESTARTS   AGE
metallb-system             pod/speaker-qcj9x                                             0/1     Unknown   23         138d
kube-system                pod/calico-node-bs46g                                         0/1     Unknown   1          27d
metallb-system             pod/speaker-cn6nx                                             0/1     Unknown   29         160d
kube-system                pod/calico-node-62x6q                                         0/1     Unknown   1          27d

sofo@k8s-test-m:~$ kubectl -n metallb-system describe pod/speaker-qcj9x
Name:         speaker-qcj9x
Namespace:    metallb-system
Priority:     0
Node:         k8s-test-m/10.14.214.40
Start Time:   Sun, 13 Dec 2020 22:26:36 +0000
Labels:       app=metallb
              component=speaker
              controller-revision-hash=69f56bb877
              pod-template-generation=1
Annotations:  prometheus.io/port: 7472
              prometheus.io/scrape: true
Status:       Running
IP:           10.14.214.40
IPs:
  IP:           10.14.214.40
Controlled By:  DaemonSet/speaker
Containers:
  speaker:
    Container ID:  containerd://d56653f8ee10a407bd8383ebc2b0c9735a09bb6beba2ae9c49ec0500ce08a506
    Image:         metallb/speaker:v0.8.2
    Image ID:      docker.io/metallb/speaker@sha256:f1941498a28cdb332429e25d18233683da6949ecfc4f6dacf12b1416d7d38263
    Port:          7472/TCP
    Host Port:     7472/TCP
    Args:
      --port=7472
      --config=config
    State:          Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Sun, 04 Apr 2021 03:45:28 +0000
      Finished:     Sat, 01 May 2021 20:46:47 +0000
    Ready:          False
    Restart Count:  23
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      METALLB_NODE_NAME:   (v1:spec.nodeName)
      METALLB_HOST:        (v1:status.hostIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from speaker-token-gwnsh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  speaker-token-gwnsh:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  speaker-token-gwnsh
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason                  Age                    From             Message
  ----     ------                  ----                   ----             -------
  Warning  NodeNotReady            27m                    node-controller  Node is not ready
  Warning  FailedCreatePodSandBox  22m (x4 over 24m)      kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:334: copying bootstrap data to pipe caused: write init-p: broken pipe: unknown
  Warning  FailedCreatePodSandBox  21m (x8 over 24m)      kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: read init-p: connection reset by peer: unknown
  Normal   SandboxChanged          9m9s (x71 over 24m)    kubelet          Pod sandbox changed, it will be killed and re-created.
  Normal   SandboxChanged          2m22s (x9 over 3m52s)  kubelet          Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  2m22s (x9 over 3m51s)  kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: read init-p: connection reset by peer: unknown
  Warning  FailedCreatePodSandBox  43s                    kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:334: copying bootstrap data to pipe caused: write init-p: broken pipe: unknown
  Normal   SandboxChanged          1s (x9 over 94s)       kubelet          Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  1s (x8 over 93s)       kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: read init-p: connection reset by peer: unknown