ContainerOp alternative

Per request of @Bobgy:

Currently, when using ContainerOp you will get the following warning:

FutureWarning: Please create reusable components instead of constructing ContainerOp instances directly. Reusable components are shareable, portable and have compatibility and support guarantees. Please see the documentation: https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-definition-file The components can be created manually (or, in case of python, using kfp.components.create_component_from_func or func_to_container_op) and then loaded using kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.load_component_from_file
  category=FutureWarning,

The alternatives requires you to specify your ContainerOp instance as either yaml or json if you want to use it inline. This option however lacks the editor/type checking support that you would get when using the Python based ContainerOp.

Proposals:

• Offer a Python based alternative for defining your ContainerOp instance in a future proof way
• Yaml schema’s could offer editor support for the content of the definition, however it will probably be less convinient than native Python code.

[tom-dd]

There are also lightweight python components created using kfp.components.create_component_from_func.

In our case, we are coming from a “legacy” environment where we have all our pipeline code in a large container, so unfortunately we are not lightweight anymore.

Can you describe the problems you’re seeing? The loaded components are represented as strongly typed functions. When working in a Jupyter environment, the editor shows the help for the parameters and help(op) works as well. Does this work for you?

We are not using Notebooks at the moment, so I cannot judge how that would work. However, when you have pylint/mypy check your code for type safety it helps if also your ContainerOp definitions can be checked as well.

Yaml schema’s could offer editor support for the content of the definition

Can you please help me understand this request. ComponentSpec has a schema and also when you load a component from URL, file or inline text, the component structure is validated. Is there something missing?

It is validated when you load them, but having validation/autocomplete while you are writing specs is very helpful

It’s possible to construct an instance of the kfp.components.structures.ComponentsSpec class and then .save it. I’m not sure this is more convenient than lightweight python components or manual component.yaml authoring. I’d like to hear your thoughts.

I guess this is close to what could work. If I take the (a reduced) example from the link @Bobgy posted, I can identify a case however that would be caught by my editor tooling with the ContainerOp route:

component_spec = ComponentSpec(
    name='Dummy op',
    description='Dummy component for illustrative purposes',
    inputs=[
        InputSpec(name='input1', type='String'),
    ],
    implementation=ContainerImplementation(container=ContainerSpec(
        image="dummy-image",
        command=[
            "python", "runner.py",
            "--input1", InputValuePlaceholder('inpt1'),
        ],
    ))
)

vs

def slack_notification(input1: str) -> ContainerOp:
    return ContainerOp(
        name="dummy-image",
        image="dummy-image",
        arguments=[
                "python", "runner.py",
                "--input1", inpt1
        ],
    )

My editor tooling would find that inpt1 doesn’t exist

[Ark-kun]

The alternatives requires you to specify your ContainerOp instance as either yaml or json if you want to use it inline.

There are also lightweight python components created using kfp.components.create_component_from_func.

This option however lacks the editor/type checking support that you would get when using the Python based ContainerOp.

Can you describe the problems you’re seeing? The loaded components are represented as strongly typed functions. When working in a Jupyter environment, the editor shows the help for the parameters and help(op) works as well. Does this work for you?

Yaml schema’s could offer editor support for the content of the definition

Can you please help me understand this request. ComponentSpec has a schema and also when you load a component from URL, file or inline text, the component structure is validated. Is there something missing?

Offer a Python based alternative for defining your ContainerOp instance in a future proof way

It’s possible to construct an instance of the kfp.components.structures.ComponentsSpec class and then .save it. I’m not sure this is more convenient than lightweight python components or manual component.yaml authoring. I’d like to hear your thoughts.