Is there a reason why `client.images.build` doesn't stream output logs ?

Seems to be a recurrent question. I looked at the code and it seems to me there is no reason why the higher-level API generator couldn’t return the output logs as they come. This could be achieved by using a sort of fake future for the image object returned by the build function. Something like that (very rough sketch) :

    def build(self, **kwargs):
        resp = self.client.api.build(**kwargs)
        if isinstance(resp, six.string_types):
            return self.get(resp)
        last_event = None
        image_id = None
        result_stream, internal_stream = itertools.tee(json_stream(resp))
        
        result = {}

        def gen():
            for chunk in internal_stream:
                if 'error' in chunk:
                    raise BuildError(chunk['error'], result_stream)
                if 'stream' in chunk:
                    match = re.search(
                        r'(^Successfully built |sha256:)([0-9a-f]+)$',
                        chunk['stream']
                    )
                    if match:
                        image_id = match.group(2)
                    else:
                        yield chunk
                last_event = chunk
            
            if image_id:
                result['image'] = self.get(image_id)
            else:
                raise BuildError(last_event or 'Unknown', result_stream)

        return result, gen()

Maybe this could be switched-on by adding an option to build for example stream_output ?

And this can be used like this :

    image, output_iter = docker_client.images.build(fileobj=fd, tag=args.tag)

    for line in output_iter:
        print(line)
    
    print(image)

Of course the nicer solution for that would be to use asyncio but that’s then becoming more complicated 😛

[haizaar]

Hit this one as well - need to show logs as they come. Ended up using low level API: from_env().api.build()

[vladak]

The other thing is that with using the low level build API, one is basically forced to reimplement the gory-details of docker/models/images.py#build(), i.e. matching the regex so that image ID can be determined (in order to construct the Image instance) and grep-ing for error patterns. If one wants to avoid that, it is necessary to resort to ugly workarounds like this:

  try:
      image, _ = client.images.build(path=".", tag="master")
  except docker.errors.BuildError as e:
      logger.error(e)
      # Rebuild the image using low level API, hoping the failure is reproducible.
      live_log_generator = client.api.build(path=".", tag="master")
      for line in live_log_generator:
          line_dict = json.loads(line)
          if line_dict.get("stream"):
              logger.error(line_dict["stream"].rstrip())

which doubles the time needed to diagnose the problem.