client.get() on an unknown key hangs forever

Issue

Calling client.get(graph, [keys, ...]) with a key that does not exist hangs forever. The timeout argument is not respected in this case.

Expected behavior

A clear error, the threaded scheduler throws a friendly KeyError for example.

Reproducible example

import operator as op
from distributed import Client
from dask import delayed

client = Client()

result = delayed(op.add)(1, 2)
graph = result.__dask_graph__()

client.get(graph, ["does_not_exist"], timeout="1s")  # <--- hangs

Interrupting gives the following stack trace:

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-3-c3eb58815d0d> in <module>
      1 result = delayed(op.add)(1, 2)
      2 graph = result.__dask_graph__()
----> 3 client.get(graph, ["does_not_exist"], timeout="1s")

/usr/local/lib/python3.7/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2590                     should_rejoin = False
   2591             try:
-> 2592                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2593             finally:
   2594                 for f in futures.values():

/usr/local/lib/python3.7/site-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
   1888                 direct=direct,
   1889                 local_worker=local_worker,
-> 1890                 asynchronous=asynchronous,
   1891             )
   1892 

/usr/local/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    766         else:
    767             return sync(
--> 768                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    769             )
    770 

/usr/local/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    340     else:
    341         while not e.is_set():
--> 342             e.wait(10)
    343     if error[0]:
    344         typ, exc, tb = error[0]

/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py in wait(self, timeout)
    550             signaled = self._flag
    551             if not signaled:
--> 552                 signaled = self._cond.wait(timeout)
    553             return signaled
    554 

/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py in wait(self, timeout)
    298             else:
    299                 if timeout > 0:
--> 300                     gotit = waiter.acquire(True, timeout)
    301                 else:
    302                     gotit = waiter.acquire(False)

[jrbourbeau]

That’s not intentional. Improvements to documentation are definitely welcome! You can feel free to include updates to the Client.get docstring in the PR that closes this issue, or in a separate PR as it’s not directly related to the unknown key issue. Whichever option you prefer

[a-reich]

Hi dask team, is it still desired to fix this? I could try it if so.