KubeCluster loses the ability to run commands against a deployed cluster after periods of inactivity.

Cluster is created as:

cluster = KubeCluster(pod_template=worker_pod, scheduler_pod=sched_pod, deploy_mode='remote')

What happened: Initially, all API commands run against the cluster, such as ‘scale’, ‘close’, etc… function as expected. After some period of time, the cluster object fails to authenticate with the cluster and all API commands return a failed status due to being ‘unauthorized’.

site-packages/dask_kubernetes/core.py", line 81, in close
    await self.core_api.delete_namespaced_pod(name, namespace)
...
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

What you expected to happen: I would expect the cluster object to re-authenticate, or provide some mechanism to re-attach to the created cluster.

# Put your MCVE code here

Anything else we need to know?:

Environment: Environment is a GKE cluster. Prior to losing the ability to authenticate, everything else functions as expected.

• Dask version: 2.30.0
• Python version: 3.8.6
• Operating System: GKE + CoS
• Install method (conda, pip, source):

[drobison00]

@jacobtomlinson Any updates on this? If not, do you have any intuition on what’s preventing the re-auth from occurring? If I get some extra time I could take a look.

[jacobtomlinson]

Sorry I’ve been out for a while and this is currently on the backlog.

If you have some time to investigate it would be much appreciated.