Add ability to schedule inference service pods to certain nodes
See original GitHub issue/kind feature
Describe the solution you’d like Allow user to specify node selector, affinity, anti-affinity, and toleration so that they can control in which node the inference service pods will be scheduled. The implementation should also allow predictor and transformer to have a different configuration (related to https://github.com/kubeflow/kfserving/issues/666).
It could look something like this
apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
name: transformer-cifar10
spec:
default:
predictor:
pytorch:
modelClassName: Net
storageUri: gs://kfserving-samples/models/pytorch/cifar10
nodeSelector:
disktype: ssd
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: another-node-label-key
operator: In
values:
- another-node-label-value
tolerations:
- key: "key"
operator: "Exists"
effect: "NoSchedule"
transformer:
custom:
container:
image: gcr.io/kubeflow-ci/kfserving/image-transformer:latest
name: user-container
Anything else you would like to add: Knative serving has this limitation (https://github.com/knative/serving/issues/1816). Alternatively, it could be implemented in the webhook.
Issue Analytics
- State:
- Created 4 years ago
- Reactions:2
- Comments:15 (8 by maintainers)
Top Results From Across the Web
Add ability to schedule inference service pods to certain nodes
Allow user to specify node selector, affinity, anti-affinity, and toleration so that they can control in which node the inference service pods ......
Read more >Assigning Pods to Nodes - Kubernetes
You can constrain a Pod so that it is restricted to run on particular node(s), or to prefer to run on particular nodes....
Read more >Assigning Pods to Nodes - Amazon EKS Workshop
In this Chapter, we will review how the strategy of assigning Pods works, alternatives and recommended approaches. You can constrain a pod to...
Read more >Placing pods on specific nodes using node selectors
You can control which node a specific pod is scheduled on by using node selectors and labels. To use node selectors and labels,...
Read more >Overview - KServe Documentation Website
In order to achieve node scheduling, KServe leverages kubernetes ability to constrain a pod to a node (see Assigning Pods to Node). Using...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@salanki @pradithya @dalfos Looks like knative has lifted the restriction now https://github.com/knative/serving/pull/8645/files.
@yuzisun: I don’t think it’s reasonable to expect every user to implement a custom admission controller just to add affinities. It also makes sense to have them as part of the InferenceService, instead of having to define my node requirements for my InferenceService in custom code in a separate admission controller.