question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Protect KFServing endpoint

See original GitHub issue

/kind feature

Currently, a kfserving-gateway is used for kfserving inference endpoints. It uses LoadBalancer and all endpoints are publicly available without protection.

This doesn’t meet requirement for production grade cluster. We need a way to project these endpoints. Seems currently probe is a blocker to inference behind authn layer.

Describe the solution you’d like

Long term, I think there’re two options. (assume we can probe issue fixed)

  1. Remove kfserving-gateway, reuse istio-ingressgateway which means requests from external need to get authenticated. I think both IAP (GCP), Coginito/OIDC (AWS) supports programmatic authentication. I am not sure about Dex. The advantage is this solution reuse AuthN and AuthZ from existing infra.

  2. Still have separate gateway for kfserving. Leave implementation to different vendors. User could build authentication on top of it. For example, AWS can replace service with ingress and LB level authentication for it. The reason we don’t reuse istio-gateway is because we can have other authentication strategy for kfserving. For example, each user can request different APIKeys for different models. etc.

  3. Have a middleware for kfserving to manage API Keys. Not sure if there’s existing solution on Kubernetes. This sounds like very common use case.

Anything else you would like to add: Pipeline SDK will have similar issue, we can consider this together.

The user experience should be simple enough, client can get clientId and secret to refresh token or just use an assigned token to make call directly.

Solution needs to be latency optimized.

Related Issue: https://github.com/kubeflow/kfctl/issues/140 https://github.com/kubeflow/kubeflow/issues/4912

@yuzisun @ellis-bigelow @animeshsingh @jlewi @cliveseldon

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:5
  • Comments:29 (16 by maintainers)

github_iconTop GitHub Comments

4reactions
ellistarncommented, Apr 3, 2020

I would definitely lean option 1. Our default install should provide a single gateway that is configured correctly for kubeflow. Multiple gateways creates confusion and potential security holes. Option 3 seems like a non-starter. I’d prefer that we aren’t in the business of authentication. Curious to hear what others thing.

2reactions
yuzisuncommented, May 2, 2020

@jlewi @ellis-bigelow @animeshsingh @Jeffwan @cliveseldon @krishnadurai I have tested the knative probe fix with GCP/IAP, I am able to get KFServing working e2e with https://github.com/kubeflow/manifests/pull/1137, however KFServing as is does not work out of the box for following issues.

  1. currently KFServing is using host based routing and seems like GCP/IAP only support path based routing from what I have experimented, I could be wrong. If that is the case we need additional virtual service configuration to map the url path to correct service which requires work in KFServing.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kfserving-kubeflow-yuzi-dan-kfserving-test
  namespace: kubeflow-yuzi-dan
spec:
  gateways:
  - kubeflow/kubeflow-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /kfserving/kubeflow-yuzi-dan/sklearn-iris
    rewrite:
        uri: /v1/models/sklearn-iris:predict
    route:
    - destination:
        host: cluster-local-gateway.istio-system.svc.cluster.local
      headers:
        request:
          set:
            Host: sklearn-iris-predictor-default.kubeflow-yuzi-dan.svc.cluster.local
      weight: 100
    timeout: 300s
  1. Istio sidecar is automatically injected in kubeflow user namespace, so we need additional authorization policy to allow sending the request to inference service after getting authenticated, we are getting 403 currently. Also there is open issue that knative autoscaling does not work with deprecated istio rbac rules. A walk around is that user can add annotation sidecar.istio.io/inject: false on KFServing inference service, seems kubeflow tfserving example does the same trick.

As discussed in last WG meeting, @cliveseldon will help test this probe fix with Istio/Dex kfdef and @Jeffwan can help on aws kfdef. I am unsure how Istio/Dex will work with KFServing since it does not support programatic token, @krishnadurai @yanniszark might have some ideas.

Read more comments on GitHub >

github_iconTop Results From Across the Web

KServe with Kubeflow on AWS
This tutorial shows how to set up a load balancer endpoint for serving prediction requests over an external DNS on AWS. Note: The...
Read more >
KFServing, Model Monitoring with Apache Spark and a ...
In this talk, we present an open-source platform for serving and monitoring models at scale based on Kubeflow's model serving framework, KFServing, ...
Read more >
How to make an ML model inference on KFServing ... - Medium
You will deploy the rpm model in TensorFlow Serving (TFServing) to get an endpoint. You will use the endpoint to check how the...
Read more >
ibm/kfserving-controller-ppc64le - Quay.io
ibm / kfserving-controller-ppc64le ... No builds have been run for this repository. Description. No repository description has been set.
Read more >
How to Serve Machine Learning Models With TensorFlow ...
Endpoint here can be a direct user or other software.” In this tutorial, I'm going to show you how to serve ML models...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found