Pods not assigned to nodes in autoscaling worker-pool

Hi,

Thanks for your effort you are putting into this project! I was able to get pangeo running on a GCS cluster. I did pretty much everything as described in the tutorial and now my hub is running and I can log in and open a notebook. Now the problem: When I generate the dask-kubernetes-cluster with

from dask_kubernetes import KubeCluster
cluster = KubeCluster.from_yaml('../worker-template.yaml')
cluster.scale_up(5)

I just get 2 workers and as long as I wait it won’t become more. I can perform computations on those worker-pods though! When I check the pods in the cluster I can see, that the missing pods are not started due to missing cpu/memory resources. When I check the worker-pool, I can see that the resources are already allocated. So it seems, as if the autoscaling works, but the pods are somehow not distributed to the nodes?

Would be thankful for a hint!

[h4gen]

@jacobtomlinson Thank you for your answer! I thought that autoscaling was enabled, because as you can see the events said, that the upscaling was triggered. Nevertheless I recreated the cluster and now it works like charme! Thank you very much for your help!

[jacobtomlinson]

The key bit here is 0/3 nodes are available: 3 Insufficient cpu, 3 Insufficient memory..

This means you do not have enough cpu and memory on your compute nodes in your cluster to fulfill the request. The way to handle this is to scale you Kubernetes cluster up. If you are using GKE there should be an option to auto scale based on demand.