Clean up pods

I’ve noticed we’re starting to accumulate a whole load of pods which are in the Terminated: Completed state.

Ideally these should be cleaned up in the same way JupyterHub cleans up pods after the user has logged off.

[mrocklin]

Ah, I didn’t realize that this was necessary. It imagine that there are probably a few ways to do this:

1. We could do it explicitly when we terminate the pod
2. We could search for terminated pods when we start up new KubeClusters and close them.
3. Perhaps there is some timeout we can put on pods in Kubernetes to have it take care of them?

On Wed, Mar 14, 2018 at 5:44 AM, Jacob Tomlinson [e-mail] wrote:

I’ve noticed we’re starting to accumulate a whole load of pods which are in the Terminated: Completed state.

Ideally these should be cleaned up in the same way JupyterHub cleans up pods after the user has logged off.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/dask-kubernetes/issues/53, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszGdEbTgw7I4bPvWuS_a9w64b0nnwks5teOZngaJpZM4SqHGB .

[jacobtomlinson]

I’m going to close this out now as completed pods are cleaned up when a new cluster is started. There isn’t much we can do about fixing this when clusters are unexpectedly killed.