Question: Manage multiple progress bars in a multiprocessing environment

Hi,

first, thanks a lot for this beautiful library. I wanted to know if it is possible to manage a Progress object with multiple progress bars in a multiprocessing environment. I just want to have a very basic progress bar for every process. My first tries did not succeed due to the fact that the Progress object cannot be pickled (due to its usage of an RLock?). TypeError: can't pickle _thread.RLock objects.

[ChristianMichelsen]

@willmcgugan Thanks a lot for your answer. After a bit of research, debugging, and generally playing around with the multiprocessing module and got the following to work:

def worker(queue_in, queue_out):
    init = very_slow_initialisation()
    while True:
        filename = queue_in.get(block=True)
        if filename is None:
            break
        result = fast_function(filename, init)
        queue_out.put((filename, result))


def main(df):

    queue_in = Queue()
    queue_out = Queue()
    the_pool = Pool(NUM_CORES, worker, (queue_in, queue_out))

    filenames = get_filenames()
    N = len(filenames)

    with Progress() as progress: 
        task_id = progress.add_task("Task", total=N)

        for filename in filenames:
            queue_in.put(filename)

        # Get and print results
        results = []
        for _ in range(N):
            results.append(queue_out.get())
            progress.advance(task_id)

    for _ in range((N):
        queue_in.put(None)

    # prevent adding anything more to the queue and wait for queue to empty
    queue_in.close()
    queue_in.join_thread()

    # prevent adding anything more to the process pool and wait for all processes to finish
    the_pool.close()
    the_pool.join()

    return results

[jpfeuffer]

Thanks a lot. It did. It worked fine after re-structuring and using a Pipe and letting the spawned processes send their updates to the main process with their task_id.