Update cpu and memory requests after cluster creation

On our Pangeo deployment users get a default worker-template.yaml which allows them to create clusters by simply running cluster = KubeCluster() without having to worry about what a kubernetes even is.

However in some occasions people want to be able to update the memory and cpu ratios of their workers depending on what they are running. The current workflow for this is to either specify the whole template as a dict or to copy the default worker-template.yaml, understand it, update the values and then user KubeCluster.from_yaml().

Personally I ended up writing a couple of helper functions in my notebook which look like this:

def update_worker_memory(cluster, new_limit):
    cluster.pod_template.spec.containers[0].resources.limits["memory"] = new_limit
    cluster.pod_template.spec.containers[0].resources.requests["memory"] = new_limit
    if '--memory-limit' in cluster.pod_template.spec.containers[0].args:
        index = cluster.pod_template.spec.containers[0].args.index('--memory-limit')
        cluster.pod_template.spec.containers[0].args[index + 1] = new_limit
    return cluster

def update_worker_cpu(cluster, new_limit):
    cluster.pod_template.spec.containers[0].resources.limits["cpu"] = new_limit
    cluster.pod_template.spec.containers[0].resources.requests["cpu"] = new_limit
    if '--nthreads' in cluster.pod_template.spec.containers[0].args:
        index = cluster.pod_template.spec.containers[0].args.index('--nthreads')
        cluster.pod_template.spec.containers[0].args[index + 1] = new_limit
    return cluster

This allows me to adjust the worker template after the cluster has been created and all new workers will follow the updated values.

I’m considering how to add this functionality into the core project. I’m inspired by the dask-jobqueue SLURMCluster which allows you to specify cores and memory as kwargs. Therefore perhaps @mrocklin, @jhamman or @guillaumeeb have thoughts.

Before I go charging in to raise a PR I would like to discuss options.

• Would it be useful to add methods to the KubeCluster object to update sizes after creation as I am above?
• Should we add kwargs to the cluster init and if so should they create the cluster and use the helpers or update the config before creation?
• Are there any other ways of specifying memory and cpu that I haven’t captured in the examples above?

[mrocklin]

I think that this seems like a reasonable request. I think that it would require additional logic to the Adaptive class that looked at resources when making requests. It’s non-trivial work, but seems reasonably in-scope.

[guillaumeeb]

We are thinking about this in https://github.com/dask/distributed/issues/2118 and https://github.com/dask/distributed/issues/2208#issuecomment-419140500.

But we didn’t talk about adaptive part yet (“scale them according to what the client requests”), which would probably need modifications too.