[sdk] Compilation with V2_COMPATIBLE mode fails when inputPath/outputPath name contains an empty space

Environment

• KFP version:

• KFP SDK version: 1.8.13

• All dependencies version:

kfp                      1.8.13
kfp-pipeline-spec        0.1.16
kfp-server-api           1.8.3

Steps to reproduce

import kfp
from kfp.v2 import dsl
from kfp.v2.dsl import component
from kfp.v2.dsl import Artifact, Output


@component
def file_generator(path: Output[Artifact]):
    with open(path, "w") as f:
        f.write("Hello World!")

input_path_op = kfp.components.load_component_from_text("""
name: Input path consumer
inputs:
- {name: input path}
implementation:
  container:
    image: alpine
    command:
    - sh
    - -exc
    - cat $0
    - inputPath: input path
""")

output_path_op = kfp.components.load_component_from_text("""
name: output path generator
outputs:
- {name: output path}
implementation:
  container:
    image: alpine
    command:
    - sh
    - -exc
    - echo Hello World! > $0
    - outputPath: output path
""")


@dsl.pipeline(name='my-pipeline')
def my_pipeline():
    file_generator_task = file_generator()
    input_path_op(file_generator_task.outputs["path"])
    # Comment this line out when you want to test the behavior of a component with outputs
    # output_path_op()


kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
    pipeline_func=my_pipeline,
    package_path='pipeline.yaml')

---

Run the code and we will get the following error.

% python3 pipeline.py
/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py:79: UserWarning: V2_COMPATIBLE execution mode is at Beta quality. Some pipeline features may not work as expected.
  warnings.warn('V2_COMPATIBLE execution mode is at Beta quality.'
Traceback (most recent call last):
  File "/Users/y-nishioka/repos/test_kfp/pipeline.py", line 48, in <module>
    kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1175, in compile
    self._create_and_write_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1227, in _create_and_write_workflow
    workflow = self._create_workflow(pipeline_func, pipeline_name,
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1058, in _create_workflow
    workflow = self._create_pipeline_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 786, in _create_pipeline_workflow
    templates = self._create_dag_templates(pipeline, op_transformers)
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 751, in _create_dag_templates
    v2_compat.update_op(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/v2_compat.py", line 158, in update_op
    "metadataPath": op.input_artifact_paths[artifact_name],
KeyError: 'input path'

op.input_artifact_paths is {'input-path': '/tmp/inputs/input_path/data'} and artifact_name is input path so we’re getting this error.

---

In the case of output_path_op, we will get the following error.

% python3 pipeline.py
/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py:79: UserWarning: V2_COMPATIBLE execution mode is at Beta quality. Some pipeline features may not work as exp
ected.
  warnings.warn('V2_COMPATIBLE execution mode is at Beta quality.'
Traceback (most recent call last):
  File "/Users/y-nishioka/repos/test_kfp/pipeline.py", line 48, in <module>
    kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1175, in compile
    self._create_and_write_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1227, in _create_and_write_workflow
    workflow = self._create_workflow(pipeline_func, pipeline_name,
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1058, in _create_workflow
    workflow = self._create_pipeline_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 786, in _create_pipeline_workflow
    templates = self._create_dag_templates(pipeline, op_transformers)
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 751, in _create_dag_templates
    v2_compat.update_op(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/v2_compat.py", line 187, in update_op
    "metadataPath": op.file_outputs[artifact_name],
KeyError: 'output path'

op.file_outputs is {'output-path': '/tmp/outputs/output_path/data'} and artifact_name is output path so we’re getting this error.

Expected result

Compilation succeeds.

v2 SDK doc says

Input and output names are converted to Pythonic names (spaces and symbols are replaced with underscores and letters are converted to lowercase). For example, an input named Input 1 is converted to input_1.

So using a name with empty spaces should be allowed in v2.

Materials and Reference

---

Impacted by this bug? Give it a 👍.

[surajkota]

@AlexandreBrown no, you cannot use standalone deployment instructions to upgrade KFP when using Kubeflow. KFP deployed with a Kubeflow installation is multi user flavour which depends on other components like Istio, cert-manager etc. Kubeflow-1.4 installs KFP-1.7. @zijianjoy to use v2 and address this issue, user would have to install v2.0-alpha correct?

Unfortunately as of today I dont think there is clear guidance on Kubeflow upgrade or if you can just upgrade KFP component in particular Kubeflow versions

[zijianjoy]

@surajkota That is right, for using V2 feature, you need to start using KFP v2.0.0-alpha or above.

In general, you maintain a copy of Kubeflow manifests locally, whenever you want to upgrade a kubeflow component, like KFP. You can compare the diff between old version and new version, and use tools like kpt, helm chart, etc. to perform upgrade.