weakref.finalize called when kubernetes already cleaned up the key_file

When a python app terminates, https://github.com/dask/dask-kubernetes/blob/ccb9864d463a3e14988af53a947a658cf32e7e10/dask_kubernetes/core.py#L690 this finalize() is called. Here we construct a new kubernetes client, however if it uses a key file in /tmp/key_file format (it creates a temp file “extract” for the urllib3 package, because it’s embedded in the kube config file) then it’s not available at this point anymore. It was cleaned up when kubernetes module run it’s shutdown procedure. This raises exceptions.

How can we somehow call the finalize manually before the script terminates (and check in the actual finalize call whether it was run or not)? I couldn’t trigger this code path from the script 😕 del client, del cluster, gc.collect(), client.close(), cluster.close(), cluster.scheduler.close() didn’t work, I couldn’t get rid if the instance (which can imply a memory leak too)

[jacobtomlinson]

Thanks for raising this, and agreed.

We try to clean up resources with finalize to help folks who forget to clean up their cluster. However it looks like we may have lost the ability to do this regularly.

We construct a new client because finalize is called after the asyncio event loop is closed, so no async code will work. So we have to construct a new sync client.

My preference here would be to implement cluster.close() correctly. But also maintain a codepath to try and clean up on finalize if it hasn’t been done manually.

One benefit of having .close implemented correctly is we can also use a context manager. Today this would work, but fall back to finalize to clean up.

from dask.distributed import Client
from dask_kubernetes import KubeCluster

with KubeCluster.from_yaml('worker-spec.yml') as cluster:
    with Client(cluster) as client:
        # Do some Dask work

[jacobtomlinson]

Closing due to inactivity