async Client with sync LocalCluster

When a Client is passed a LocalCluster and the asynchronous=True flag, it silently ignores the explicit flag and gets it from the cluster instead.

>>> cluster = await distributed.LocalCluster(asynchronous=True)
>>> client = await distributed.Client(cluster, asynchronous=True)
>>> client.asynchronous
True
>>> cluster = distributed.LocalCluster()
>>> client = await distributed.Client(cluster, asynchronous=True)
>>> client.asynchronous
False
>>> cluster = distributed.LocalCluster()
>>> client = await distributed.Client(cluster.scheduler.address, asynchronous=True)
>>> client.asynchronous
True

The same happens for an eventual explicit loop parameter. Both issues are caused by these lines, which cause Client.loop to be forced to match Client.cluster.loop: https://github.com/dask/distributed/blob/9f91653398aee681ba0010b1053f3bd1b415894b/distributed/client.py#L647-L653

I see in the code how a few things will break if the client and client.cluster end up having two different event loops. Not sure what the correct solution would be?

[mrocklin]

Yes, I think that erring loudly in this case, perhaps with the workaround that you suggested with the address would be a good solution.

On Mon, May 11, 2020 at 8:51 AM crusaderky [e-mail] wrote:

I don’t have a use case for two event loops, but when prototyping on jupyter notebook, if I need particular worker settings, I typically find it useful to write cluster and worker on two separate lines. If the client is asynchronous, i frequently forget to set the asynchronous flag on the cluster.

Should I simply implement a check that, if address is a cluster, then the loop and asynchronous parameters must either be omitted or match those of the cluster?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/3789#issuecomment-626788546, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTAIK3NQHSV7HEEMT4DRRANHTANCNFSM4M5XRIXQ .

[lmeyerov]

FWIW, we’re seeing the reverse need in GPU land: sync client to an async cluster.

– Most of our code is async client <> async cluster – Many libs (ex: blazingsql) assume sync client, which seems to require a sync cluster – We don’t want 2 clusters, as dask_cudf / RMM takes ~300MB baseline per instance – … so we’d like to occasionally generate a sync client to an existing async cluster