[feature] default to the emissary executor

Update - 2021.9.13

edited by @Bobgy

KFP 1.7.0 is released. documentation is live: https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/ choosing an executor will be an option starting from KFP 1.7.

Update - 2021.8.6

edited by @Bobgy

Discussed with @kramachandran and decided to delay the default change for at least one KFP minor version, because argo emissary executor is still in Alpha state. There can be other corner cases that we haven’t found, so we should give people some time to test it out.

Therefore, I’d like to propose the following:

• still default to docker executor in 1.7.0 release
• provide emissary executor as an option and document how to migrate
• collect feedback from people who starts to try emissary executor

TODOs:

• https://github.com/kubeflow/website/pull/2857
• Make emissary available in installations
  • #6294
  • Make env/platform-agnostic-emissary
• #6252
• Upgrade kfp-ci test infra to argo v3.1.1 https://github.com/kubeflow/testing/commit/03c625828ac9cc13283bd226371280b0f5704c09

Issues discovered when testing argo v3.1.0 with KFP:

• https://github.com/kubeflow/pipelines/issues/5930
• https://github.com/kubeflow/pipelines/issues/5937 (fixed in v3.1.1)
• https://github.com/kubeflow/pipelines/issues/5944 (fixed in v3.1.1)
• https://github.com/argoproj/argo-workflows/issues/6238

=== the following are the original proposal ===

What feature would you like to see?

Update argo to 3.11 such that we can use the emissary executor as decided by @bobgy in https://github.com/kubeflow/pipelines/pull/4645#issuecomment-845691359

What is the use case or pain point?

The Docker, Kubelet, PNS and K8sapi executors have severe limitations that will be solved by Argo 3.1 and the emissary executor.

Here is an overview https://argoproj.github.io/argo-workflows/workflow-executors/

In a nutshell docker breaks security completely and is incompatible with kubernetes 1.19 https://github.com/kubeflow/pipelines/issues/5714 Kubernetes is also moving to containerd anyway and a lot of users want proper non-docker support for a long time https://github.com/kubeflow/pipelines/issues/1654

Kubelet needs some configuration and has the same limitation as k8sapi and rootless PNS: “Output artifacts must be saved on volumes (e.g. emptyDir) and not the base image layer (e.g. /tmp)” so it breaks the leightweight python components if there is no volume mounted or the output directory is inside the base layer https://github.com/kubeflow/pipelines/pull/4645

One can use a PNS with root rights (PTRACE and CHROOT) to circumvent this, but this obviously breaks the security again.

So only the emissary executor https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary is feasible which does more ore less what i proposed in https://github.com/kubeflow/pipelines/pull/4645 and what was started by Argos @alexec in https://github.com/argoproj/argo-workflows/pull/4766

The Argo update has been done before in https://github.com/kubeflow/pipelines/pull/5266 and https://github.com/kubeflow/pipelines/pull/4693 by @xinbinhuang @Bobgy

Is there a workaround currently?

# Emmissary executor, needs argo 3.1+
# Sadly visualizations are broken(26.04.2021) for k8sapi and emissary

kubectl -n kubeflow patch configmap/workflow-controller-configmap --type='json' -p='[{"op": "replace", "path": "/data/containerRuntimeExecutor", "value": "emissary"}]'

kubectl edit deployment/workflow-controller -n kubeflow
...
    spec:
      containers:
      - env:
        - name: "LEADER_ELECTION_DISABLE"
       	  value: true
      - args:
	- --configmap
        - workflow-controller-configmap
        - --executor-image
        - docker.io/argoproj/argoexec:latest
        command:
        - workflow-controller
        image: docker.io/argoproj/workflow-controller:latest    
        env:
	- name: "LEADER_ELECTION_DISABLE"
          value: "true"
...
kubectl -n kubeflow patch configmap/workflow-controller-configmap --type='json' -p='[{"op": "replace", "path": "/data/containerRuntimeExecutor", "value": "emissary"}]'
kubectl rollout restart deployment workflow-controller -n kubeflow

or manually patching the sdk and using k8sapi as done in https://github.com/kubeflow/pipelines/pull/4645

Or use kfp-tekton

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[NikeNano]

Have started the work to upgrade should be easy to go to the next version when it is out.

[juliusvonkohout]

Is there any estimation of when the emissary is going to be the default executed?

I think there are some small runasbonnroot bugs for V2 pipelines, but V1 and V2-Compatible should be fine. Feel free to help there… So I hope that they change the default to emissary in 1.9. (kubeflow 1.5 most likely) . I bet that @Bobgy wants to update argo argo to >= 3.2.2 to get the most mature emissary executor. There is also an emissary feedback issue somewhere here where you can ask.