Allow to start a Scheduler in a batch job

One of the goal of ClusterManager object is to be able to launch a remote scheduler. In dask-jobqueue scope, this probably means submitting a job which will start a Scheduler, and then connect to it.

We probably still lacks some remote interface between ClusterManager and scheduler object for this to work, so it will probably mean to extend APIs upstream.

Identified Scheduler method to provide:

• retire_workers(n, minimum, key)
• scheduler_info(), already existing, see if sufficient,
• add_diagnostic_plugin(), and mostly retrive plugin information remotely

I suspect that adaptive will need to change significantly too, this will maybe lead to having a transitional adaptive logic in dask-jobqueue, and other remote function to add in scheduler.

This is in the scope of #170.

[manuel-rhdt]

In practice I doubt that the scheduler will be expensive enough that system administrators will care. They all ask about this, but I don’t think that it will be important in reality.

Another reason to support this is for networking rules. In some systems (10-20%?) compute nodes are unable to connect back to login nodes. So here placing the scheduler on a compute node, and then connecting to that node from the client/login node is nice.

It may be though that this is a frequently requested but not actually useful feature.

I am currently using dask-joblib on a PBS cluster and running the scheduler on the login node. It is indeed a bit problematic because the login node has only 2gb of memory and it quickly runs out if I am not careful with the size of computation graphs.

So I think I would definitely benefit from this feature.

[lesteve]

@muammar I see that you have commented in https://github.com/dask/dask-jobqueue/pull/390#issuecomment-603558844. Could you please explain the admin rules that are in place on your cluster just to get an idea what you are allowed to do on your cluster.

You may be interested by my answer above: https://github.com/dask/dask-jobqueue/issues/186#issuecomment-568265386. Let me try to some up:

1. launch the scheduler (i.e. create the Cluster object) in an interactive job: probably easier. If you like to work in a Jupyter environment, this is doable this way. There are a few hoops to jump through (mostly SSH port forwarding to open your Jupyter notebook in your local browser at localhost:<some-port>).
2. launch the scheduler (i.e. create the Cluster object) in a batch job. Only Python scripts not Jupyter environment.
3. Starting the Dask worker by launching Dask commands yourself: https://docs.dask.org/en/latest/setup/hpc.html#using-a-shared-network-file-system-and-a-job-scheduler

In both 1. and 2. you need to bear in mind that as soon as your scheduler job finishes, you will lose all your workers after ~60s. That may mean losing the result of lenghty computations.