Allow adding a callback to a list of futures

As discussed on gitter I’m looking for a way to asynchronously handle the result of a parallel calculation represented by a list of futures e.g. to log either the success or failure of each task.

The current api allows you to chain calls and will automatically pass the result of the tasks to the next function. If one of the tasks fails you get an exception in the calling process which prevents you handling the error asynchronously - you have to block in the calling process to handle the error

The concurrent.futures api has a Future.add_done_callback method which may be worthwhile implementing in dask.distributed

[mrocklin]

You need to use the underscore version, _wait, which is a tornado coroutine rather than a blocking function. This requires understanding of Tornado coroutines and asynchronous programming. Here is a good link:

http://www.tornadoweb.org/en/stable/coroutine.html

Example

In [1]: from distributed import Client

In [2]: c = Client()

In [3]: from distributed.client import _wait

In [4]: from tornado import gen

In [5]: def div(x, y):
   ...:     return x / y
   ...:

In [6]: future = c.submit(div, 1, 0)

In [7]: @gen.coroutine
   ...: def print_status(future):
   ...:     yield _wait(future)
   ...:     print(future.status)
   ...:

In [8]: c.loop.add_callback(print_status, future)
 error

[dhirschfeld]

I found a solution which works but is pretty ugly. I’m revisiting this problem again shortly so will open new issues if there are still some rough edges in the api.

My thoughts are to try and use the new async client and as such I’m interested in the discussion around dask/distributed#1663.