question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add ability to schedule inference service pods to certain nodes

See original GitHub issue

/kind feature

Describe the solution you’d like Allow user to specify node selector, affinity, anti-affinity, and toleration so that they can control in which node the inference service pods will be scheduled. The implementation should also allow predictor and transformer to have a different configuration (related to https://github.com/kubeflow/kfserving/issues/666).

It could look something like this

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  name: transformer-cifar10
spec:
  default:
    predictor:
      pytorch:
        modelClassName: Net
        storageUri: gs://kfserving-samples/models/pytorch/cifar10
      nodeSelector:
        disktype: ssd
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/e2e-az-name
                operator: In
                values:
                - e2e-az1
                - e2e-az2
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: another-node-label-key
                operator: In
                values:
                - another-node-label-value
      tolerations:
      - key: "key"
        operator: "Exists"
        effect: "NoSchedule"
    transformer:
      custom:
        container:
          image: gcr.io/kubeflow-ci/kfserving/image-transformer:latest
          name: user-container

Anything else you would like to add: Knative serving has this limitation (https://github.com/knative/serving/issues/1816). Alternatively, it could be implemented in the webhook.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:2
  • Comments:15 (8 by maintainers)

github_iconTop GitHub Comments

4reactions
yuzisuncommented, Jul 20, 2020

@salanki @pradithya @dalfos Looks like knative has lifted the restriction now https://github.com/knative/serving/pull/8645/files.

4reactions
salankicommented, Mar 28, 2020

@yuzisun: I don’t think it’s reasonable to expect every user to implement a custom admission controller just to add affinities. It also makes sense to have them as part of the InferenceService, instead of having to define my node requirements for my InferenceService in custom code in a separate admission controller.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Add ability to schedule inference service pods to certain nodes
Allow user to specify node selector, affinity, anti-affinity, and toleration so that they can control in which node the inference service pods ......
Read more >
Assigning Pods to Nodes - Kubernetes
You can constrain a Pod so that it is restricted to run on particular node(s), or to prefer to run on particular nodes....
Read more >
Assigning Pods to Nodes - Amazon EKS Workshop
In this Chapter, we will review how the strategy of assigning Pods works, alternatives and recommended approaches. You can constrain a pod to...
Read more >
Placing pods on specific nodes using node selectors
You can control which node a specific pod is scheduled on by using node selectors and labels. To use node selectors and labels,...
Read more >
Overview - KServe Documentation Website
In order to achieve node scheduling, KServe leverages kubernetes ability to constrain a pod to a node (see Assigning Pods to Node). Using...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found