Compiler work items

This is an “umbrella issue” to capture planned work items related to the TektonCompiler:

• Generate a Pipeline with Tasks instead of a Task with steps to allow parallel execution (PR #17)
• Allow passing output parameters as inputs for subsequent Tasks (issue #19, PR #27)
• Migrate to Tekton 0.11 to make use of API v1beta1 (PR #27)
• Add Tekton integration tests (issue #28)
• Add support for dsl.VolumeOp (#51)
• Add support for set_image_pull_secrets (#54)
• Add support for conditions (#33)
• Add support for static LoopArguments (#67)
• Add support for dynamic LoopArguments (#82)
• Add support for ParallelFor (#67)
• Option to generate Tekton PipelineRun spec (#62)
• Add support for Tekton Workspaces to share resources between Tasks (#74)
• Handler for checking Argo specific params, find equivalent for Tekton (#78 #80)
• Fix big data passing (#63)
• …

Specific template conversions:

Existing code that was copied from the KFP SDK compiler but temporarily commented out. See _op_to_template.py):

• Add support for dsl.ResourceOp (#49)
• Add support for artifacts parameters (#64)
• Add support for affinity (#65) (requires PipelineRun #62)
• Add support for initContainers (#50)
• Add support for metadata pod_annotations (#76)
• Add support for metadata pod_labels (#76)
• Add support for node selector (#65) (requires PipelineRun #62)
• Add support for retries (#48)
• Add support for sidecars (#42)
• Add support for timeout (#46)
• Add support for tolerations (#65) (requires PipelineRun #62)
• Add support for volumes (#34)

In terms of prioritization, we could look at how often these features are used in any of the KFP pipeline examples (see KFP samples test report).

Of the 88 dsl.pipeline examples and samples found in the kubeflow/pipelines repository 54 fail:

  45 'volumes' are not (yet) implemented
  38 `dsl.ResourceOp` is not yet implemented
   4 `input artifacts` are not yet implemented
   3 'sidecars' are not (yet) implemented
   3 'retries' is not (yet) implemented
   1 'tolerations' is not (yet) implemented
   1 'timeout' is not (yet) implemented
   1 'nodeSelector' is not (yet) implemented
   1 'affinity' is not (yet) implemented

Update on KFP testdata compilation failure reasons as of Apr 3, 2020:

FAILURE: basic_no_decorator.py           - needs ExitHandlerOp, support pipeline with no decorator 
FAILURE: compose.py                      - needs function to flatten nested pipeline 
FAILURE: input_artifact_raw_value.py     - needs S3 compatible artifact passing
FAILURE: param_substitutions.py          - needs dsl.ResourceOp
FAILURE: resourceop_basic.py             - needs dsl.ResourceOp
FAILURE: volume_snapshotop_rokurl.py     - needs dsl.VolumeSnapshotOp
FAILURE: volume_snapshotop_sequential.py - needs dsl.VolumeSnapshotOp
FAILURE: volumeop_basic.py               - needs dsl.VolumeOp
FAILURE: volumeop_dag.py                 - needs dsl.VolumeOp
FAILURE: volumeop_parallel.py            - needs dsl.VolumeOp
FAILURE: volumeop_sequential.py          - needs dsl.VolumeOp
FAILURE: loop_over_lightweight_output.py - needs Tekton dynamic looping
FAILURE: withparam_global.py             - needs Tekton dynamic looping
FAILURE: withparam_global_dict.py        - needs Tekton dynamic looping
FAILURE: withparam_output.py             - needs Tekton dynamic looping
FAILURE: withparam_output_dict.py        - needs Tekton dynamic looping

Mapping of features from Argo to Tekton:

General guidance to map KFP/Argo features to Tekton features can be found in the KFP, Argo and Tekton Comparision Spreadsheet maintained by @afrittoli

[Tomcli]

Hi @ckadner, can you add the below tasks as well. There are more tasks in the spreadsheet, but i think we should prioritize the basic functionalities before dive in deeper to the harder ones.

• Add support for dsl.VolumeOp
• Handler for checking Argo specific params
• Add support for set_image_pull_secrets
• Add support for passing artifacts.
• Add support for conditions

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.92

Please mark this comment with 👍 or 👎 to give our bot feedback! Links: app homepage, dashboard and code for this bot.