Uber Issue: KFServing admission hook causing widespread issues because its a global admission hook

/kind bug

We are getting lots of reports about problems caused because the KFServing admission hook is unavailable preventing pods from being created. The error message looks like the following

4m58s       Warning   FailedCreate                   replicaset/activator-5484756f7b          Error creating: Internal error occurred: failed calling webhook "inferenceservice.kfserving-webhook-server.pod-mutator": Post https://kfserving-webhook-server-service.kubeflow.svc:443/mutate-pods?timeout=30s: service "kfserving-webhook-server-service" not found

Here’s my understanding

• Currently AdmissionHooks can not be scoped by label; so a pod admission hook is being applied to all pods

• The KFServing Admission Hooks is being applied to all pods and then in the hook itself it checks whether the pod belongs to a KFServing resource and if it does applies the hook

• However, if the KFServing web hook deployment is unavailable pod creation can be blocked

• For a variety of reasons we are reaching into a deadlock state where

  • The WebHook is defined but the deployment for the hook is not defined so calls to the admission hook will fail
  • Pod creation now fails because the webhook is not defined

[yuzisun]

@maganaluis We need to use object selector on the mutating webhook configuration so that only kfserving labelled pods go through the KFServing pod mutator, the problem is that object selector is only supported kubernetes 1.15+ while kubeflow’s minimal requirement is still kubernetes 1.14. If you are on kubernetes 1.15+ you can use following command to solve the issue.

kubectl patch mutatingwebhookconfiguration inferenceservice.serving.kubeflow.org --patch '{"webhooks":[{"name": "inferenceservice.kfserving-webhook-server.pod-mutator","objectSelector":{"matchExpressions":[{"key":"serving.kubeflow.org/inferenceservice", "operator": "Exists"}]}}]}'

[jlewi]

Possible fixes

1. Add the label control-plane to the kubeflow namespace

   kubectl label namespace kubeflow control-plane=true
   

1. Change the namespaceSelector to be opt in; match namespaces with specific labels

    * This won't work because the changes will be overwritten when the controller restarts because the controller creates the webhook

Ref: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector

Possible Work Arounds

• Add the label control-plane to the kubeflow namespace
• Update the inferservice webhook to change the namespace selector to be opt in.

A possible recipe

1. Get the inference spec

   kubectl -n kubeflow get MutatingWebHookConfiguration inferenceservice.serving.kubeflow.org -o yaml > /tmp/inferceservice.yaml
   

2. Change the matchSelector

   namespaceSelector:
     matchLabels:
        serving.kubeflow.org: "true"
   

3. Apply it

   kubectl apply -f /tmp/inferenceservice.yaml
   

4. Label any namespaces in which you want to use KFServing as

   kubectl label namespace ${NAMESPACE} serving.kubeflow.org=true