Is path-based serving supported?

/kind feature

Hello everyone!

I see that the current way to serve KServe models is host-based. That is, the URL of a model looks something like this:

• http://<model-name>-<namespace>.<domain>.

This way, one needs to create a DNS record and a certificate for the sub-domain <model-name>-<namespace> where the model is hosted.

So, my question is: is there an alternative way? A path-based way?.

In that case, one should not bother about DNS records and certificates. The URL of the model could look something like this:

• http://<domain>/<namespace>/<model-name>

I couldn’t find any issues with a similar question. Is something like this currently supported?

If not, then what are the difficulties of such a feature?

[yuzisun]

@kandrio Thanks for proposal! Overall it looks great, a few details might need to pay attention:

• There are two different names: InferenceService name and model name, sometimes they can be different as a service can host multiple models, so we need to encode both service name and model name in the url path.
• There are two inference protocols kserve currently supports: v1 and v2, the endpoint paths are slightly different between the two, for example the predict path for v1 is v1/models/<model-name>:predict while for v2 it is v2/models/<model-name>/infer. The switch for v1 or v2 can be based on the protocolVersion field on InferenceService spec.

The virtual service template might look like following:

- headers:
    request:
      set:
        Host: <service-name>-<component>-default.<namespace>.svc.cluster.local
  match:
  - authority:
      regex: <host-template>
    gateways:
    - <public-gateway>
    uri:
      prefix: /<namespace>/<service-name>/<protocol-version>/models/<model-name>
  rewrite:
    uri: /<protocol-version>/models/<model-name>
  route:
  - destination:
      host: <local-gateway-service>
      port:
        number: 80
    weight: 100

[lizzzcai]

Hi @kandrio , thanks for the proposal, that is a feature we needed for a while. You can find our use cases and requirement here.

Currently, we are using an additional virtual service to route the path-based request to the cluster-local-gateway. See the example below. It will be great that KServe can support it natively.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: service1-path-based-endpoint
  namespace: user1-ns1
spec:
  gateways:
  - istio-system/ingress-gateway
  hosts:
  - api.example.com
  http:
  - name: "service1-path-based-routes"
    match:
    - uri:
        prefix: "/user1-ns1/service1/"
      ignoreUriCase: true
    rewrite:
      uri: "/"
    route:
    - destination:
        host: local-gateway.istio-system.svc.cluster.local
        port:
          number: 80
      headers:
        request:
          set:
            Host: service1.user1-ns1.svc.cluster.local
      weight: 100

Instead of urlTemplate, maybe having an additional pathTemplate to work with the existing domainTemplate to define the path will be better?

domainTemplate: {{` "{{.Name}}-{{.Namespace}}.{{.Domain}}" `}}
pathTemplate: {{` "{{.Namespace}}/{{.Name}}" `}} # user need to avoid conflict of the path by themselves.

Above will generate an inferenceService URL with the following format:

"{{.Name}}-{{.Namespace}}.{{.Domain}}/{{.Namespace}}/{{.Name}}"
# example: svc1-ns-1.example.com/ns1/svc1

If the pathTemplate is empty, then it will be host-based routing.

What do you think?