Conditional gather?

If I would like to gather the result of parallel computations that have finished (not those that have thrown an error, or are still in computation), I would write cod that does this:

scores = dict()
for assay, score_future in scores_futures.items():
    if score_future.status == 'finished':
        scores[assay] = client.gather(score_future)

Would it be of interest, and would it also make sense or not (I’m open to this not being a sensible thing), to have this wrapped in a client.gather_finished(futures) class method? Some of my parallel computations take a long time, while others finish up pretty quickly, so being able to gather just the finished jobs and continue prototyping, while allowing myself to come back later and gather everything later, might be of use, I think.

[eriknw]

The as_completed object is able to be used in a non-blocking fashion. For example:

my_tasks = as_completed([x, y, z])

# option 1, the batch will be empty if none are ready.  Doesn't block
batch = my_tasks.next_batch(block=False)

# option 2, check if any are ready
if not my_tasks.queue.empty():
    batch = my_tasks.next_batch()
    # or you could just grab one result if that fits better in your loop

Hope this helps. The docs could maybe be improved. Check out the source code:

http://distributed.dask.org/en/latest/_modules/distributed/client.html#as_completed

[martindurant]

Yes, I suppose you are right. Your loop is simple enough, and seems it works well - I don’t suppose it’s designed to be run too often.

There may be interest in supplying this convenience method. I would recommend proposing a PR with clear docstring and tests (some futures don’t finish for a long time, some error).