No workers error when trying to scatter

I am trying to scatter data to workers but getting a TimeoutError: No workers found. I can do sacct and see my workers running, despite dask throwing this error. Here is a minimal working example which produces the error on my machine:

from dask_jobqueue import SLURMCluster
from dask.distributed import Client
cluster = SLURMCluster(cores=1,memory="19 GB",interface='ib0')
cluster.scale(2)
client = Client(cluster)
[future] = client.scatter([1.0], broadcast=True)

[crusaderky]

I’m facing this problem too. The proposed workaround above has the big issue that keeps a copy of all the data on the scheduler, which quickly balloons into the gigabytes for me.

[danpf]

The way I’ve handled this is by submitting, not scattering.

future = client.submit(list, [1.0])

seems to work well enough.