How to yield result instead of getting the result list?

This is my code

import time
from joblib import Parallel, delayed
def producer():
    with open("some_large_file") as f:
        for line in f:
            yield line

def func(i):
    time.sleep(1)  # or other time comsumption operations
    return i
out = Parallel(n_jobs=10, verbose=100)(delayed(func)(i) for i in producer())
print(out)

My purpose is to read a large file (about 10G)，and do some operations on each line. The final result will be saved in the out variable, which is a list object stored in memory. Can joblib yield immediate results during run jobs? Then I can write the result content to a file rather than store them in the memory?

[GaelVaroquaux]

This feature is quite challenging to code (danger of race conditions, in particular when dealing with exceptions).

There is a pull request in progress on this feature: https://github.com/joblib/joblib/pull/588 We hope to merge in soonish, but these things are tricky.

[npyoung]

Second this feature request. Often my parallel jobs have large outputs that I want to process and write to disk as they become available rather than keep them around in memory until all jobs have completed. I would use something like multiprocessing.Pool.imap for this but I need the advanced pickling and memmap conversion of joblib.