Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Handle pods in Unknown state

See original GitHub issue

When nodes in a cluster misbehave, they sometimes leave pods in an Unknown state:

jupyter-user1                   0/1       Unknown   0          3h        10.244.28.17    k8s-pool1-12345678-12
jupyter-user2                   0/1       Unknown   1          5h        10.244.28.8     k8s-pool1-12345678-12
jupyter-user3                   0/1       Unknown   1          4h        10.244.28.11    k8s-pool1-12345678-12
jupyter-user4                   1/1       Unknown   0          6h        <none>          k8s-pool1-12345678-12
jupyter-user5                   0/1       Unknown   0          3h        10.244.28.14    k8s-pool1-12345678-12
jupyter-user6                   0/1       Unknown   1          3h        10.244.28.15    k8s-pool1-12345678-12
jupyter-user7                   0/1       Unknown   1          4h        10.244.28.9     k8s-pool1-12345678-12

I can invoke delete on the pods but they don’t actually go away. The pods are only cleared when the node has been rebooted – stopping the node is insufficient.

Can KubeSpawner work around this? Or should it even try, give that kubernetes seems to be at fault here?

Issue Analytics

State:
Created 6 years ago
Comments:8 (5 by maintainers)

Top GitHub Comments

1reaction

chicocvenanciocommented, Mar 25, 2019

But because this is likely an issue with the cluster, deleting a pod that is Unknown should probably come with some extra warning output about the state of the pod before deleting it, to help with diagnostics.

Sounds good. I’ll try to start working on this next week.

1reaction

minrkcommented, Mar 25, 2019

But because this is likely an issue with the cluster, deleting a pod that is Unknown should probably come with some extra warning output about the state of the pod before deleting it, to help with diagnostics.