Resource Manager Interface

Resource Manager Interface

The first step is hard, but standing up is harder.

Deploying distributed on a remote cluster is a large hurdle between the curious user and serious adoption. Efforts to improve deployment can significantly improve community uptake and value.

Unfortunately, there are a large number of deployment solutions so there is no single piece of code we can write to satisfy most potential users. We have direct relations with people who employ the following deployment technologies:

1. YARN
2. Mesos
3. SLURM
4. Torque
5. SGE
6. SSH
7. Local processes

Supporting each of these is doable in isolation, but when we consider supporting all in aggregate we desire a common framework.

Additionally, near-future versions of the distributed scheduler may want to dynamically request and release worker resources depending on load. Providing a single resource management interface to the scheduler would allow integration with several technologies in a sane and maintainable manner.

Question: What does such a resource management interface look like?

Example Interface

Let us consider the following signals from scheduler to resource manager (RM) and back.

• Scheduler -> RM: Please give me more workers
• Scheduler -> RM: I no longer need these workers
• RM -> Scheduler: I plan to take back this worker very soon
• RM -> Scheduler: This worker has unexpectedly died

We could implement this as a Python ResourceManger object that has fuctions for “Please give me more workers” and “I no longer need these workers” that the scheduler can call as well as callbacks that the scheduler provides on ResourceManager creation for “I plan to take back this worker soon” and “this worker has unexpectedly died”.

class ResourceManager(object):
    def __init__(self, on_reclaiming_worker, on_worker_died, **kwargs):
        ...

    def request_workers(self, nworkers, **kwargs):
        ...

    def release_workers(self, worker_ids):
        ...

We might then subclass this interface for processes, SSH, Mesos, Yarn, etc…

Questions

Different resource management systems employ different information. Yarn allows the specification of CPUs and Memory and the use of containers, HPC job schedulers have fixed job durations, Mesos operates a bit differently from all.

The example operations provided above may not fit all resource managers. Some may provide only a subset of this functionality while some may have a superset. How do we balance the needs to specialize to a particular system while also maintaining compatibility with many systems and keeping code simple.

cc @hussainsultan @broxtronix @quasiben @danielfrg

[ellisonbg]

Subscribing to the discussion…

[ogrisel]

Support for SLURM and SGE can be implemented at once by reusing the lightweigth abstraction provided by https://github.com/clusterlib/clusterlib.