Add ability to schedule inference service pods to certain nodes

/kind feature

Describe the solution you’d like Allow user to specify node selector, affinity, anti-affinity, and toleration so that they can control in which node the inference service pods will be scheduled. The implementation should also allow predictor and transformer to have a different configuration (related to https://github.com/kubeflow/kfserving/issues/666).

It could look something like this

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  name: transformer-cifar10
spec:
  default:
    predictor:
      pytorch:
        modelClassName: Net
        storageUri: gs://kfserving-samples/models/pytorch/cifar10
      nodeSelector:
        disktype: ssd
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/e2e-az-name
                operator: In
                values:
                - e2e-az1
                - e2e-az2
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: another-node-label-key
                operator: In
                values:
                - another-node-label-value
      tolerations:
      - key: "key"
        operator: "Exists"
        effect: "NoSchedule"
    transformer:
      custom:
        container:
          image: gcr.io/kubeflow-ci/kfserving/image-transformer:latest
          name: user-container

Anything else you would like to add: Knative serving has this limitation (https://github.com/knative/serving/issues/1816). Alternatively, it could be implemented in the webhook.

[yuzisun]

@salanki @pradithya @dalfos Looks like knative has lifted the restriction now https://github.com/knative/serving/pull/8645/files.

[salanki]

@yuzisun: I don’t think it’s reasonable to expect every user to implement a custom admission controller just to add affinities. It also makes sense to have them as part of the InferenceService, instead of having to define my node requirements for my InferenceService in custom code in a separate admission controller.