Dask backend returns early if no workers allocated

If the cluster has no workers, the dask backend returns immediately.

In [1]: from dask.distributed import Client, LocalCluster

In [2]: cluster = LocalCluster(n_workers=0)

In [3]: client = Client(cluster)

In [4]: import joblib

In [5]: with joblib.parallel_backend('dask'):
   ...:     results = joblib.Parallel(verbose=100)(joblib.delayed(lambda x: x**2)(x) for x in range(10))
   ...:
[Parallel(n_jobs=-1)]: Using backend DaskDistributedBackend with 0 concurrent workers.
[Parallel(n_jobs=-1)]: Done   0 out of   0 | elapsed:    0.0s finished

In [6]: import dask

In [7]: joblib.__version__
Out[7]: '0.13.0'

In [8]: dask.__version__
Out[8]: '1.0.0'

In this case the backend should probably hang until a worker is added, or error immediately. This is complicated since the number of workers could change dynamically during computation. I’m not sure what’s best here.

xref https://github.com/dask/dask-yarn/issues/46

[ogrisel]

I just encountered this bug myself today on a script with dask-jobqueue…

Maybe erroring by default is a good idea but we should also have an option to be able to wait for the first worker to be provisioned.

[ogrisel]

For the record, this issue is now fixed in joblib 0.13.1.