[FR] Default resource requirement/limits for the KFP UI and system services

UPDATE: at the end, we decided to only add resource requirements, see discussion in https://github.com/kubeflow/pipelines/issues/5236#issuecomment-790301148

It’s desirable to provide a set of default resource requirement & limits for KFP UI & system services, to make sure their QoS is Guaranteed by default.

https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/ I’m not exactly sure what will be reasonable, because if they are set too low, the services may stop operating when there are workloads reaching a limit. But setting them to make QoS Guaranteed is also important, because otherwise when there are many other workloads, KFP UI & API services may be evicted because default QoS is BestEffort and BestEffort Pods are the first to be evicted by Kubernetes when it runs out of resources.

[NikeNano]

According to the argo documentation the memory and cpu usage for argo scales linearly with the nbr of workflows, see. So users will probably have to adjust this according if they are running heavier workloads or like to reduce costs.

I would be happy to update this!

/assign

[Bobgy]

Got some help from Sid Palas:

A couple of example request settings:
ml-pipeline (api server)
        requests:
          cpu: '2'
          memory: 4Gi
ml-pipeline-ui
        requests:
          cpu: 10m
          memory: 70Mi
workflow-controller (argo)
        requests:
          cpu: 200m
          memory: 3Gi
minio
          requests:
            cpu: 20m
            memory: 25Mi
persistent-agent
          requests:
            cpu: 120m
            memory: 2Gi

see thread https://kubeflow.slack.com/archives/CE10KS9M4/p1613655024114300