Cache Webhook "disabled" by default in Azure

What happened:

[A clear and concise description of what the bug is.] Right now, mutating webhooks are used for components such as the cache-server. Previously there was an issue with the knative webhook, so the label “control-plane” was attached to prevent the webhook from triggering all the time. (Refer to https://github.com/kubeflow/kubeflow/issues/4511). However, Azure by default adds the below namespace selector to mutatingwebhooks to prevent applying to AKS internal namespaces. (https://github.com/Azure/AKS/issues/1771)

namespaceSelector:
    matchExpressions:
    - key: control-plane
      operator: DoesNotExist

As the KF namespace comes with “control-plane: kubeflow”, this causes the cache server to fail to mutate any pods in Kubeflow.

What did you expect to happen:

It seems unfair to expect Kubeflow to fix this issue, as this dependency is inherently caused by Azure upstream. Perhaps we can update the Azure docs / default deploy to tell the users that these components won’t work as intended?

Environment:

Azure

How did you deploy Kubeflow Pipelines (KFP)?

KFP version: 1.2

KFP SDK version: 1.4

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

[andre-lx]

In azure, the problem is that istio pods are not being injected to the pods in kubeflow.

Azure enforces that the MutatingWebhookConfiguration - istio-sidecar-injector in kubeflow is being automatically edited by AKS to add the following match expression in the namespaceSelector:

matchExpressions:
- key: control-plane
  operator: DoesNotExist

So the MutatingWebhookConfiguration looks like this:

namespaceSelector:
	matchExpressions:
    - key: control-plane
      operator: DoesNotExist
	matchLabels:
  		istio-injection: enabled

This will exclude the kubeflow namespace, since the namespace have the label:

labels:
    control-plane: kubeflow

To solve this issue, you need to deactivate the admission enforcer from aks, using the following annotation in the MutatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    admissions.enforcer/disabled: "true"
  name: istio-sidecar-injector

I believe, that the following issues are all related to this, so you don’t need to disable istio (changing the DestinationRules from ISTIO_MUTUAL to DISABLE):

https://github.com/kubeflow/pipelines/issues/4469 https://github.com/kubeflow/kubeflow/issues/5271 https://github.com/kubeflow/kubeflow/issues/5277 https://github.com/Azure/AKS/issues/1771

+info: https://docs.microsoft.com/en-us/azure/aks/faq#can-i-use-admission-controller-webhooks-on-aks

[andre-lx]

Hi @danishsamad. Not sure, actually.

In my case, deleting the match expression didn’t solve the issue, since AKS automatically add that match expression.

I’m not sure if I did the manual delete or if the annotation deleted it.

Maybe I added the annotation and eliminated the match expression manually.

Anyway, in new clusters, the annotation worked as expected -> the match expression is not added and the istio pods are injected to the pipelines pods.