[FEA] Dynamic configuration of worker device memory limits

I’d like to be able to dynamically configure worker device memory limits. Given a running cluster, a user may have various sized workloads. Some workloads would benefit from device memory limits, while others would be harmed by this. Requiring an explicit cluster bringdown/bringup to change memory limits can sometimes pose a real challenge. Allowing a user to dynamically configure a device memory limit would let them optimally run each individual workload.

From discussions with @pentschev , this would likely require changing the LRU parameters and require that the cache be empty. Given that, I wonder if this would be possible in something like a call to Client.restart, though of course not requiring that might be desirable, too.

[pentschev]

Thanks for confirming @beckernick . In the interest of keeping maintenance costs down to a minimum, I’ll go ahead and close the PR and this issue. If in the future this becomes important again, we can pickup from where we left.

[beckernick]

This is no longer relevant for the original motivating use case. I suspect that @madsbk 's work in #353 and his ongoing work to understand the details of the underlying device memory while spilling may further reduce the need for this.

With that said, it might be a nice small addition in the future.