Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unable to curl the predictions for any of the kfserving samples in docs.

See original GitHub issue

/kind bug

What steps did you take and what happened: Running the kubeflow V1.0 with minikube installation. I am just following the sample examples of kfserving for tensorflow and sklearn from the docs: this is one of the yaml file that I am using

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  default:
    predictor:
      minReplicas: 1
      sklearn:
        storageUri: "gs://kfserving-samples/models/sklearn/iris"

On doing the curl request after setting INGRESS HOST,PORT and SERVICE_NAME, I am getting-

 curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d $INPUT_PATH
* About to connect() to 192.168.99.106 port 31380 (#0)
*   Trying 192.168.99.106...
* Connected to 192.168.99.106 (192.168.99.106) port 31380 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: sklearn-iris.kubeflow.example.com
> Content-Length: 36
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 36 out of 36 bytes
< HTTP/1.1 404 Not Found
< content-length: 139
< content-type: text/html; charset=UTF-8
< date: Mon, 01 Feb 2021 14:12:02 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 238
<
* Connection #0 to host 192.168.99.106 left intact
<html><title>404: Model with name sklearn-iris does not exist.</title><body>404: Model with name sklearn-iris does not exist.</body></html>

I have already set changed the knative-gateway to kubeflow-gateway which was the issue in version 1.0.2. When I kubectl exec into the kfserving-container pod I don’t find the directory /mnt/models where the model should have been I guess as per the logs file. What did you expect to happen: One thing that I observed was that in some of the github issues there were init:containers as well for storage provisioning purpose, I don’t see that anywhere in my deployment, I have added the logs file as well down below. I expected to get in return the predictions when I curl to the models.

Anything else you would like to add:

kubectl get inferenceservice -n kubeflow
NAME                     URL                                                                                   READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
kfserving-custom-model   http://kfserving-custom-model.kubeflow.example.com/v1/models/kfserving-custom-model   True    100                                3h28m
sklearn-iris             http://sklearn-iris.kubeflow.example.com/v1/models/sklearn-iris                       True    100                                138m
sklearn-reg              http://sklearn-reg.kubeflow.example.com/v1/models/sklearn-reg                         True    100                                10m

kubectl describe inferenceservice sklearn-iris -n kubeflow
Name:         sklearn-iris
Namespace:    kubeflow
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"sklearn-iris","namespace":"ku...
API Version:  serving.kubeflow.org/v1alpha2
Kind:         InferenceService
Metadata:
  Creation Timestamp:  2021-02-01T11:32:49Z
  Generation:          5
  Resource Version:    1159041
  Self Link:           /apis/serving.kubeflow.org/v1alpha2/namespaces/kubeflow/inferenceservices/sklearn-iris
  UID:                 7ee098da-9689-4afe-bc97-1d682b9a4898
Spec:
  Default:
    Predictor:
      Min Replicas:  1
      Sklearn:
        Resources:
        Storage Uri:  gs://kfserving-samples/models/sklearn/iris
Status:
  Canary:
  Conditions:
    Last Transition Time:  2021-02-01T11:34:15Z
    Status:                True
    Type:                  DefaultPredictorReady
    Last Transition Time:  2021-02-01T11:34:15Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2021-02-01T11:34:15Z
    Status:                True
    Type:                  RoutesReady
  Default:
    Predictor:
      Host:  sklearn-iris-predictor-default.kubeflow.example.com
      Name:  sklearn-iris-predictor-default-62jm7
  Traffic:   100
  URL:       http://sklearn-iris.kubeflow.example.com/v1/models/sklearn-iris
Events:      <none>

SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kubeflow -o jsonpath='{.status.url}' | cut -d "/" -f 3)
echo $SERVICE_HOSTNAME
sklearn-iris.kubeflow.example.com

kubectl describe deployment sklearn-iris-predictor-default-62jm7-deployment -n kubeflow
Name:                   sklearn-iris-predictor-default-62jm7-deployment
Namespace:              kubeflow
CreationTimestamp:      Mon, 01 Feb 2021 17:02:52 +0530
Labels:                 app=sklearn-iris-predictor-default-62jm7
                        serving.knative.dev/configuration=sklearn-iris-predictor-default
                        serving.knative.dev/configurationGeneration=1
                        serving.knative.dev/revision=sklearn-iris-predictor-default-62jm7
                        serving.knative.dev/revisionUID=cdb5087c-360e-40bb-9d86-7c66a66352d4
                        serving.knative.dev/service=sklearn-iris-predictor-default
                        serving.kubeflow.org/inferenceservice=sklearn-iris
Annotations:            autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
                        autoscaling.knative.dev/minScale: 1
                        autoscaling.knative.dev/target: 1
                        deployment.kubernetes.io/revision: 1
                        internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/sklearn/iris
                        queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
                        serving.knative.dev/creator: system:serviceaccount:kubeflow:default
Selector:               serving.knative.dev/revisionUID=cdb5087c-360e-40bb-9d86-7c66a66352d4
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:       app=sklearn-iris-predictor-default-62jm7
                serving.knative.dev/configuration=sklearn-iris-predictor-default
                serving.knative.dev/configurationGeneration=1
                serving.knative.dev/revision=sklearn-iris-predictor-default-62jm7
                serving.knative.dev/revisionUID=cdb5087c-360e-40bb-9d86-7c66a66352d4
                serving.knative.dev/service=sklearn-iris-predictor-default
                serving.kubeflow.org/inferenceservice=sklearn-iris
  Annotations:  autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
                autoscaling.knative.dev/minScale: 1
                autoscaling.knative.dev/target: 1
                internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/sklearn/iris
                queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
                serving.knative.dev/creator: system:serviceaccount:kubeflow:default
                traffic.sidecar.istio.io/includeOutboundIPRanges: *
  Containers:
   kfserving-container:
    Image:      gcr.io/kfserving/sklearnserver@sha256:2c6e7cd36ffb0efd47ed5a1fa8b2e338b36558719c3fb74d6b6ff3e9d69da9f6
    Port:       8080/TCP
    Host Port:  0/TCP
    Args:
      --model_name=sklearn-iris
      --model_dir=/mnt/models
      --http_port=8080
    Environment:
      PORT:             8080
      K_REVISION:       sklearn-iris-predictor-default-62jm7
      K_CONFIGURATION:  sklearn-iris-predictor-default
      K_SERVICE:        sklearn-iris-predictor-default
    Mounts:
      /var/log from knative-var-log (rw)
   queue-proxy:
    Image:       gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:792f6945c7bc73a49a470a5b955c39c8bd174705743abf5fb71aa0f4c04128eb
    Ports:       8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Requests:
      cpu:      25m
    Readiness:  exec [/ko-app/queue -probe-period 0] delay=0s timeout=10s period=1s #success=1 #failure=3
    Environment:
      SERVING_NAMESPACE:                      kubeflow
      SERVING_SERVICE:                        sklearn-iris-predictor-default
      SERVING_CONFIGURATION:                  sklearn-iris-predictor-default
      SERVING_REVISION:                       sklearn-iris-predictor-default-62jm7
      QUEUE_SERVING_PORT:                     8012
      CONTAINER_CONCURRENCY:                  0
      REVISION_TIMEOUT_SECONDS:               60
      SERVING_POD:                             (v1:metadata.name)
      SERVING_POD_IP:                          (v1:status.podIP)
      SERVING_LOGGING_CONFIG:                 {
                                                "level": "info",
                                                "development": false,
                                                "outputPaths": ["stdout"],
                                                "errorOutputPaths": ["stderr"],
                                                "encoding": "json",
                                                "encoderConfig": {
                                                  "timeKey": "ts",
                                                  "levelKey": "level",
                                                  "nameKey": "logger",
                                                  "callerKey": "caller",
                                                  "messageKey": "msg",
                                                  "stacktraceKey": "stacktrace",
                                                  "lineEnding": "",
                                                  "levelEncoder": "",
                                                  "timeEncoder": "iso8601",
                                                  "durationEncoder": "",
                                                  "callerEncoder": ""
                                                }
                                              }
      SERVING_LOGGING_LEVEL:
      SERVING_REQUEST_LOG_TEMPLATE:
      SERVING_REQUEST_METRICS_BACKEND:        prometheus
      TRACING_CONFIG_BACKEND:                 none
      TRACING_CONFIG_ZIPKIN_ENDPOINT:
      TRACING_CONFIG_STACKDRIVER_PROJECT_ID:
      TRACING_CONFIG_DEBUG:                   false
      TRACING_CONFIG_SAMPLE_RATE:             0.100000
      USER_PORT:                              8080
      SYSTEM_NAMESPACE:                       knative-serving
      METRICS_DOMAIN:                         knative.dev/internal/serving
      USER_CONTAINER_NAME:                    kfserving-container
      ENABLE_VAR_LOG_COLLECTION:              false
      VAR_LOG_VOLUME_NAME:                    knative-var-log
      INTERNAL_VOLUME_PATH:                   /var/knative-internal
      SERVING_READINESS_PROBE:                {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
      ENABLE_PROFILING:                       false
      SERVING_ENABLE_PROBE_REQUEST_LOG:       false
    Mounts:                                   <none>
  Volumes:
   knative-var-log:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   sklearn-iris-predictor-default-62jm7-deployment-5c997778d (1/1 replicas created)
Events:          <none>

This is the logs that I see for the containers: For kfserving-container-

{"log":"[I 210201 13:41:26 storage:43] Copying contents of /mnt/models to local\n","stream":"stderr","time":"2021-02-01T13:41:26.426888759Z"
{"log":"[I 210201 13:41:26 kfserver:96] Listening on port 8080\n","stream":"stderr","time":"2021-02-01T13:41:26.463778184Z"}
{"log":"[I 210201 13:41:26 kfserver:98] Will fork 1 workers\n","stream":"stderr","time":"2021-02-01T13:41:26.514102548Z"}
{"log":"[W 210201 13:50:36 web:2243] 404 POST /v1/models/sklearn-reg:predict (127.0.0.1) 10.96ms\n","stream":"stderr","time":"2021-02-01T13:

For queue-proxy-

{"log":"{\"level\":\"info\",\"ts\":\"2021-02-01T13:41:24.595Z\",\"caller\":\"logging/config.go:108\",\"msg\":\"Successfully created the logg
{"log":"{\"level\":\"info\",\"ts\":\"2021-02-01T13:41:24.598Z\",\"caller\":\"logging/config.go:109\",\"msg\":\"Logging level set to info\"}\
{"log":"{\"level\":\"info\",\"ts\":\"2021-02-01T13:41:24.606Z\",\"logger\":\"queueproxy\",\"caller\":\"metrics/exporter.go:99\",\"msg\":\"Fl
{"log":"{\"level\":\"info\",\"ts\":\"2021-02-01T13:41:24.606Z\",\"logger\":\"queueproxy\",\"caller\":\"metrics/prometheus_exporter.go:37\",\
{"log":"{\"level\":\"info\",\"ts\":\"2021-02-01T13:41:24.606Z\",\"logger\":\"queueproxy\",\"caller\":\"metrics/exporter.go:108\",\"msg\":\"S

Environment:

Istio Version:
Knative Version:
KFServing Version:
Kubeflow version: v1.0.2
Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm] k8s_istio.v1.0.2
Minikube version: v1.15.1
Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

OS (e.g. from /etc/os-release):

NAME="Oracle Linux Server"
VERSION="7.6"
ID="ol"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Oracle Linux Server 7.6"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:6:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.6
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.6

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:6 (3 by maintainers)

Top GitHub Comments

2reactions

yuzisuncommented, Feb 4, 2021

@chinmay22897 just noticed you were deploying in kubeflow namespace, this is expected to not work as kubeflow is control plane namespace and you are not supposed to deploy application there.

0reactions

yuzisuncommented, Feb 4, 2021

It is documented here https://github.com/kubeflow/kfserving#kfserving-with-kubeflow-installation

Top Results From Across the Web

Unable to curl the predictions for any of the kfserving samples ...

Running the kubeflow V1.0 with minikube installation. I am just following the sample examples of kfserving for tensorflow and sklearn from the ...

MLflow Models — MLflow 2.1.0 documentation

Each MLflow Model is a directory containing arbitrary files, ... and valid model outputs (e.g. model predictions generated on the training dataset).

Serving service from kfserving github examples created with ...

It turned out that my local docker registry wasn't visible from kubernetes. kubectl get events shows InternalError "Unable to fetch image .

Deploying to KFServing — BentoML documentation

This guide demonstrates how to serve a scikit-learn based iris classifier model with BentoML on a KFServing cluster. The same deployment steps are...

End-to-end Pipeline with KFServing - D2iQ Help Center

Because each component will be packaged as a Docker image. The base image must therefore contain all dependencies. Any dependencies you install manually...