fit something with parallel_backend is unreasonably slow

I set up a Dask cluster and tries to reproduce the distributed Machine Learning tutorial from https://github.com/dask/dask-tutorial/blob/master/08_machine_learning.ipynb when I run the command

%%time
with joblib.parallel_backend("dask", scatter=[X, y]):
    grid_search.fit(X, y)

it’s not complete even after an hour and compare to the fitting without Dask as a backend I thought it’s unreasonably slow. Since it does not give me an error message I wonder if anybody here could give me some advice. Thanks

[jakirkham]

Shouldn’t be implemented a warning for such case?

Please see PR ( https://github.com/dask/distributed/pull/2627 ).

[TomAugspurger]

As noted in https://joblib.readthedocs.io/en/latest/auto_examples/parallel/distributed_backend_simple.html#sphx-glr-auto-examples-parallel-distributed-backend-simple-py,

This example shows the simplest usage of the dask distributed backend, on the local computer.

This is useful for prototyping a solution, to later be run on a truly distributed cluster, as the only change to be made is the address of the scheduler.

So you’ve created a dask.distributed “cluster” locally on your machine. But to make it useful you need to connect to a real cluster (setup described in https://docs.dask.org/en/latest/setup.html)