Progress bars for steps (integration of tqdm)

Is there a way to use tqdm inside a step (especially a foreach task)? I want to have a progress bar for each parallel task. Currently I am only able to see progress bars when a task has already successfully finished.

I am not familiar how logging is handled in metaflow but here are some examples from tqdm that could be helpful to make the progress bar work: https://github.com/tqdm/tqdm/blob/master/examples/parallel_bars.py https://github.com/tqdm/tqdm/blob/master/examples/redirect_print.py

In general, it looks like messages are only printed when the task completed. This makes the use of tqdm pointless. Is there a way to immediately print to console?

Here is the source code for the example from the gif:

from metaflow import FlowSpec, step


class HelloFlow(FlowSpec):
    """
    A flow where Metaflow prints 'Hi'.

    Run this flow to validate that Metaflow is installed correctly.

    """
    @step
    def start(self):
        """
        This is the 'start' step. All flows must have a step named 'start' that
        is the first step in the flow.

        """
        print("HelloFlow is starting.")
        self.multi_processing = list(range(4))
        self.next(self.hello, foreach="multi_processing")

    @step
    def hello(self):
        """
        A step with parallel processing that should be monitored with tqdm.

        """

        from tqdm import tqdm
        from time import sleep
        from random import random

        interval = random() * 0.001
        for _ in tqdm(range(10000)):
            sleep(interval)

        self.next(self.join)

    @step
    def join(self, inputs):
        """
        Join our parallel branches and merge results.

        """

        self.next(self.end)

    @step
    def end(self):
        """
        This is the 'end' step. All flows must have an 'end' step, which is the
        last step in the flow.

        """

        print("HelloFlow is all done.")


if __name__ == '__main__':
    HelloFlow()

[crk-codaio]

Sorry for not having an update on this. We are looking into it internally and seeing if there is a way we can make this work. From the screencast you’ve pasted, clearly the log lines from tqdm are:

1. Getting buffered (and hence seeing it later)
2. Not seeing updates of progress - this is the one I need to spend more time debugging if there is a way to make it work with our log buffering + multiprocessing solution.

[crk-codaio]

Thank you for the very elaborate description (with screencasts 😃) Will take a look and get back to you.