Wait for workers to join before continuing

Moved from dask/dask-kubernetes#87

It would be useful to have the cluster.scale() method block until the workers join. This was originally raised in dask-kubernetes but is applicable to dask-jobqueue and dask-yarn so would make more sense to implement here.

Suggestions from @mrocklin:

One solution here might be to use SchedulerPlugins to explicitly trigger events whenever a new worker reaches the scheduler. Calling wait on the cluster object might register a tornado.locks.Event or tornado.locks.Condition that gets triggered when the right number of workers have arrived.

@jhamman implemented a SchedulerPlugin for dask-jobqueue in order to improve adaptive operation (he wanted adaptive to be aware of pending jobs). This work happened in dask/dask-jobqueue#63 and might be the sort of thing we could think about generalizing. Any thoughts on this @jhamman?

[mrocklin]

I think it would make sense to have an explicit wait method or wait= keyword to scale that would allow a user to opt-in to blocking semantics. I would use this for benchmarking.

[mrocklin]

In case someone wanted to use wait from within an asynchronous environment.

async with SGECluster(...) as cluster:
    cluster.scale(10)
    await cluster.wait()
    x.compute()