Can't use Queues w/ `processes=False`

Goal: Share a few small queues between tasks under various Client configurations.

Issue: Using distributed 1.22.0,

from dask.distributed import Client, Queue

client = Client(processes=False)
q = Queue()

def put():
    q.put(55)

def get():
    print(q.get())

fut = client.submit(put)
res = client.submit(get)

results in

Traceback (most recent call last):
  File ".../python3.6/site-packages/distributed/queues.py", line 261, in __setstate__
    assert client.address == address
AttributeError: 'Client' object has no attribute 'address'

...

File ".../python3.6/site-packages/distributed/comm/core.py", line 178, in _raise
    raise IOError(msg)
OSError: Timed out trying to connect to 'inproc://10.0.0.41/16785/7' after 10 s: connect() didn't finish in time

This works as intended when processes=True. I also can’t use queue.Queue because it won’t be pickleable, so I was hoping dask.distributed.Queue would allow me to work around that.

[mrocklin]

Possibly resolved in https://github.com/dask/distributed/pull/2221

This is also the sort of thing for which it would be nice to have an explicit Semaphore object. There is https://github.com/dask/distributed/issues/2007 if you ever feel like diving into the guts of the distributed scheduler. It’s a decent entry issue.

[mrocklin]

Replicated. Thanks for the minimal bug report Chris. I’ll take a look.

I’m also curious, what is your intended application here?

On Wed, Aug 29, 2018 at 1:30 PM, Chris White [e-mail] wrote:

Goal: Share a few small queues between tasks under various Client configurations.

Issue: Using distributed 1.22.0,

from dask.distributed import Client, Queue

client = Client(processes=False) q = Queue() def put(): q.put(55) def get(): print(q.get())

fut = client.submit(put) res = client.submit(get)

results in

Traceback (most recent call last): File “…/python3.6/site-packages/distributed/queues.py”, line 261, in setstate assert client.address == addressAttributeError: ‘Client’ object has no attribute ‘address’ …

File “…/python3.6/site-packages/distributed/comm/core.py”, line 178, in _raise raise IOError(msg)OSError: Timed out trying to connect to ‘inproc://10.0.0.41/16785/7’ after 10 s: connect() didn’t finish in time

This works as intended when processes=True. I also can’t use queue.Queue because it won’t be pickleable, so I was hoping dask.distributed.Queue would allow me to work around that.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/2220, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszDUDbynET6_FSB54riE7L8JiWy2Rks5uVs_AgaJpZM4WSAGM .