make joblib.Parallel return a generator

Often one wants to perform simple operations on the output of a very long sequence of tasks. If the number of outputs is large, it may be inefficient or impossible to store them in a list. Instead, add functionality to joblib.Parallel so that one can do:

parallel_job = ( delayed( job )( param ) for param in so_many_job_params )  # generator for input
for output in Parallel(n_jobs=10, iterable=parallel_job):                  # generator as output 
   do_something( output )

In the example above, I’ve added the job iterable to the constructor of Parallel. The only required change would be to add an __iter__(self) method to Parallel which has almost identical functionality to __call__(self.iterable), but instead uses self.iterable and yields an element one completed job at a time, rather than returning a list of outputs.

[beasteers]

I would love this functionality to be added - if not just so you could wrap Parallel with a progress bar. I see there are a bunch of closed PRs trying to implement this, but none of them have been merged for some reason 😕

In the meantime, if anyone wants a quick drop-in (albeit hacky) solution, this has been working for me. It works without having to copy and make deep edits to the original code.

import joblib
from joblib import delayed
import threading

class Parallel(joblib.Parallel):
    def it(self, iterable):
        try:
            t = threading.Thread(target=self.__call__, args=(iterable,))
            t.start()

            i = 0
            output = self._output
            while t.is_alive() or (output and i < len(output)):
                # catch the list reference and store before it's overwritten
                if output is None:
                    output = self._output
                # yield when a new item appears
                if output and i < len(output):
                    yield output[i]
                    i += 1
        finally:
            t.join()


########
# usage
########

import time
import tqdm

def task(x):
    time.sleep(1)
    return x * 2

xs = range(12)
it = Parallel(3).it(delayed(task)(x) for x in xs)
pbar = tqdm.tqdm(it, total=len(xs))
for x in pbar: 
    pbar.write(str(x))

[cgarciae]

I coded this in a library called pypeln, however it uses multiprocessing and some libraries on OSX crash because of this so I have to fallback to joblib for that OS for some work I am doing.