Option to report progress while building context

When building an exceptionally large context, there is no way to report progress. I’d opt to add an option to the util’s make context method of progress=False, so that people can manually call the function and set progress=True if they desire it.

To see the problem in action, use this repo (note the docker branch) and run toolset/run-tests.py --install server --docker --test '' (you will need python 2.7 and the requirements listed here. Building teh context can sometimes take up to 15 minutes (note: this is on an NFS filesystem which is notoriously slow when tarring a lot of small files) and normally takes about 5 minutes (on my local SSD)

[speedplane]

I monkey patched docker/api/build.py and progress info via the class below. This class wraps a file object and will report how much of has been read. Once this class is defined, you can see progress by adding context = StreamPrinter(context) before we make the _post call.

class StreamPrinter(object):
    '''
    Wrap a file object and print out read progress. Used in conjunction with
    the 
    '''
    def __init__(self, file):
        self.read_so_far = 0
        self.starttime = datetime.datetime.utcnow()
        self.file = file

    def __iter__(self, *args, **kwargs):
        self.starttime = datetime.datetime.utcnow()
        return self

    def chk_print(self, fn):
        old_megs = self.read_so_far / 1024 / 1024
        out = fn()
        self.read_so_far += len(out)
        new_megs = self.read_so_far / 1024 / 1024
        if old_megs != new_megs:
            log.debug('Sent %s megabytes. Time elapsed: %s'%(new_megs,
                datetime.datetime.utcnow() - self.starttime))
        return out

    def next(self):
        return self.chk_print(fn = lambda: next(self.file))

    def read(self, *args, **kwargs):
        return self.chk_print(fn = lambda: self.file.read(*args, **kwargs))

This is still a bit messy and I’m not sure that this is the best approach, which is why I didn’t make a pull request. But thought others may find it useful.

[gabrieldemarmiesse]

https://gabrieldemarmiesse.github.io/python-on-whales/ The only downside is that it’ll be open source next year.