[DRAFT] Move adaptive scaling functionality into dedicated process

Problem With adaptive scaling being executed on the client-side, it’s easy to execute multiple (possibly mutually-exclusive) adaptive scaling policies on a single distributed cluster through the use of multiple Cluster objects.

Goal At any given moment, there should be one, and only one, adaptive scaling policy in place that is being applied to the distributed cluster (not Cluster object).

Additional benefits:

• simplifies the client-side
• allows clients to query and modify the adaptive scaling policy another one has put into place

[jacobtomlinson]

As a side-note, would it make sense to add a draft label?

Went with discussion as that seemed more fitting.

[hendrikmakait]

It’s interesting that @hendrikmakait talks about the Client a lot in the first issue, but as @fjetter says this logic lives in Cluster.

To clarify, I am saying that the execution of the adaptive scaling logic lives on the client-side, i.e., a user process (or multiple ones). @fjetter is correct in that the called methods are attached to the Cluster class.

• simplifies the client

I’ve adjusted that to the client-side.

I think the issue here is that there is some distributed cluster, i.e., a collection of workers and a scheduler, and a Cluster object.