[feature] Add hooks to transform compiled template/workflow manifests

Feature Area

/area sdk

What feature would you like to see?

Argo workflows has features that aren’t exposed in KFP, such as synchronization, template defaults, and retry expressions. If I want to use an argo feature that isn’t exposed in KFP, I can file a PR against this repo and wait for a new release, or fork or patch the sdk, or subclass the compiler and override private methods. None of these seem like good options!

Instead, it would be useful if KFP included template- and workflow-level hooks allowing users to arbitrarily transform the compiled manifests. I don’t have a strong opinion about where these hooks should go, but I think @jli and/or @maxhully suggested adding options to Compiler.compile or to PipelineContext.

What is the use case or pain point?

Use argo workflows options that aren’t yet supported in KFP.

Is there a workaround currently?

Let me know if there’s a better workaround, but all I’ve thought of so far is patching the sdk or overriding private methods.

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[chensun]

Reducing the generated workflow size seems to be orthogonal to the hook. That being said, we’re moving towards v2, where compiler no longer produce Argo yaml but some intermediate representation defined by this proto: https://github.com/kubeflow/pipelines/blob/master/api/v2alpha1/pipeline_spec.proto So I’m not sure if you would still need to a hook to modify the compilation result. By the time KFP v2 is ready, the conversion from this IR to Argo workflow yaml would happen in the backend.