[FR] Configurable Image in Component.yaml

Would it be possible to make the image name configurable in component.yaml specification, so that the end user could override the image. It is possible to write a function to modify the yaml before it is read by kfp.components.load_component_from_text, however I think this would be a useful addition to component.yaml specification, this is something that was supported by ContainerOp.

For example the following component.yaml adds another additional inputValue to specify the docker image. However this does not work when loading the component.yaml.

name: echo
description: Academic Component. Echoes input to output.
metadata:
  annotations:
    version: "0.0.1"
inputs:
  - { name: Text, type: String, description: Text to be echoed }
  - { name: Docker Image, type: String, description: Docker Image to be used, default: alpine }
outputs:
  - { name: Echoed Text, type: String, description: Echoed Text }
implementation:
  container:
    image: {inputValue: Docker Image}
    command: [sh, -c]
    args: [
        echo $0 && mkdir -p "$(dirname "$1")" &&echo $0 > $1,
        { inputValue: Text },
        { outputPath: Echoed Text}
      ]

However this does not work. Attempting to load this component gives an error.

kfp.components.load_component_from_text(manifest.read_text())

Traceback.

   kfp.components.load_component_from_text(manifest.read_text())
 File "/usr/local/lib/python3.8/site-packages/kfp/components/_components.py", line 115, in load_component_from_text
   component_spec = _load_component_spec_from_component_text(text)
 File "/usr/local/lib/python3.8/site-packages/kfp/components/_components.py", line 164, in _load_component_spec_from_component_text
   component_spec = ComponentSpec.from_dict(component_dict)
 File "/usr/local/lib/python3.8/site-packages/kfp/components/modelbase.py", line 285, in from_dict
   return parse_object_from_struct_based_on_class_init(cls, struct, serialized_names=cls._serialized_names)
 File "/usr/local/lib/python3.8/site-packages/kfp/components/modelbase.py", line 238, in parse_object_from_struct_based_on_class_init
   args[python_name] = parse_object_from_struct_based_on_type(value, param_type)
 File "/usr/local/lib/python3.8/site-packages/kfp/components/modelbase.py", line 158, in parse_object_from_struct_based_on_type
   raise TypeError('\n'.join(exception_lines))
TypeError: Error: ContainerImplementation.from_dict(struct=OrderedDict([('container', OrderedDict([('image', OrderedDict([('inputValue', 'Docker Image')])), ('command', ['sh', '-c']), ('args', ['echo $0 && mkdir -p "$(dirname "$1")" &&echo $0 > $1', OrderedDict([('inputValue', 'Text')]), OrderedDict([('outputPath', 'Echoed Text')])])]))])) failed with exception:
Error: ContainerSpec.from_dict(struct=OrderedDict([('image', OrderedDict([('inputValue', 'Docker Image')])), ('command', ['sh', '-c']), ('args', ['echo $0 && mkdir -p "$(dirname "$1")" &&echo $0 > $1', OrderedDict([('inputValue', 'Text')]), OrderedDict([('outputPath', 'Echoed Text')])])])) failed with exception:
Error: Structure "OrderedDict([('inputValue', 'Docker Image')])" is incompatible with type "<class 'str'>". Structure is not the instance of the type, the type does not have .from_dict method and is not generic.
Error: GraphImplementation.from_dict(struct=OrderedDict([('container', OrderedDict([('image', OrderedDict([('inputValue', 'Docker Image')])), ('command', ['sh', '-c']), ('args', ['echo $0 && mkdir -p "$(dirname "$1")" &&echo $0 > $1', OrderedDict([('inputValue', 'Text')]), OrderedDict([('outputPath', 'Echoed Text')])])]))])) failed with exception:
__init__() got an unexpected keyword argument 'container'
Error: Structure "OrderedDict([('container', OrderedDict([('image', OrderedDict([('inputValue', 'Docker Image')])), ('command', ['sh', '-c']), ('args', ['echo $0 && mkdir -p "$(dirname "$1")" &&echo $0 > $1', OrderedDict([('inputValue', 'Text')]), OrderedDict([('outputPath', 'Echoed Text')])])]))])" is not None.
Error: Structure "OrderedDict([('container', OrderedDict([('image', OrderedDict([('inputValue', 'Docker Image')])), ('command', ['sh', '-c']), ('args', ['echo $0 && mkdir -p "$(dirname "$1")" &&echo $0 > $1', OrderedDict([('inputValue', 'Text')]), OrderedDict([('outputPath', 'Echoed Text')])])]))])" is incompatible with type "typing.Union[kfp.components._structures.ContainerImplementation, kfp.components._structures.GraphImplementation, NoneType]" - none of the types in Union are compatible.

[Ark-kun]

A mutable image tag does not give reproducibility.

Fair point. But the same is also true about components. For reproducibility you need a strict component version linking to a strict container image version.

it’s also possible to replace the container image after creating an instance of the component. I was not aware that this is possible. Could you point me to the relevant function or variable ?

What instantiate a component, you get a task object. The task object is ContainerOp, which has .container property which allows you to configure any Kubernetes container properties (image, resources, etc).

So, you can tweak the container image after instantiating the component:

task1 = op1(...)
task1.container.image = ...

Does this work for you?

Following is something I have resorted to.

For a CI/CD scenario the best way would be to create a new version of component.yaml every time you create a new container version. This way, the pipeline can link to a strict component version and have reproducibility.

P.S. I had a thought about a building a compiler feature which allows substituting container images. I wonder whether this would be useful.

[munagekar]

task1 = op1(...)
task1.container.image = ...

Does this work for you?

This seems reasonable and elegant solution for component users who have specific requirements. The scenarios I mentioned are rare and may be it is not worthwhile to build a compiler feature for this, especially when it is possible to override the image trivially.