sklearn-iris example doesn't have any related activity with Kfserving on Openshift

Hi,

I installed kfserving following the tutorial on https://github.com/kubeflow/kfserving/blob/master/docs/OPENSHIFT_GUIDE.md#test-kfserving-installation Then i tried to run the example sklearn-iris, after i apply the yaml from https://github.com/kubeflow/kfserving/blob/master/docs/samples/v1beta1/sklearn/v1/sklearn.yaml

No activities or errors shown for the sample.

oc get inferenceservices sklearn-iris -n kfserving-test
NAME           URL   READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION   AGE
sklearn-iris                                                                               54m

oc describe inferenceservices sklearn-iris -n kfserving-test
Name:         sklearn-iris
Namespace:    kfserving-test
Labels:       <none>
Annotations:  API Version:  serving.kubeflow.org/v1beta1
Kind:         InferenceService
Metadata:
  Creation Timestamp:  2021-04-19T21:04:34Z
  Finalizers:
    inferenceservice.finalizers
  Generation:  1
  Managed Fields:
    API Version:  serving.kubeflow.org/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:predictor:
          .:
          f:sklearn:
            .:
            f:storageUri:
    Manager:      oc
    Operation:    Update
    Time:         2021-04-19T21:04:23Z
    API Version:  serving.kubeflow.org/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:spec:
        f:predictor:
          f:containers:
          f:sklearn:
            f:args:
            f:image:
      f:status:
        .:
        f:components:
          .:
          f:predictor:
    Manager:         manager
    Operation:       Update
    Time:            2021-04-19T21:04:51Z
  Resource Version:  119867624
  Self Link:         /apis/serving.kubeflow.org/v1beta1/namespaces/kfserving-test/inferenceservices/sklearn-iris
  UID:               678000b2-ba71-4184-bf15-c05911db1916
Spec:
  Predictor:
    Sklearn:
      Name:              kfserving-container
      Protocol Version:  v1
      Resources:
        Limits:
          Cpu:     1
          Memory:  2Gi
        Requests:
          Cpu:          1
          Memory:       2Gi
      Runtime Version:  v0.5.0-rc2
      Storage Uri:      gs://kfserving-samples/models/sklearn/iris
Status:
  Components:
    Predictor:
Events:  <none>

oc logs kfserving-controller-manager-0 -c manager -n kfserving-system
{"level":"info","ts":1618866274.390503,"logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"sklearn-iris"}
{"level":"info","ts":1618866274.592222,"logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"kfserving-test","isvc":{"sklearn":{"storageUri":"gs://kfserving-samples/models/sklearn/iris","runtimeVersion":"v0.5.0-rc2","protocolVersion":"v1","name":"kfserving-container","resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}}}
I0419 21:04:35.693448       1 request.go:621] Throttling request took 1.008025253s, request: GET:https://172.21.0.1:443/apis/operators.coreos.com/v1?timeout=32s
I0419 21:04:45.694319       1 request.go:621] Throttling request took 6.185864391s, request: GET:https://172.21.0.1:443/apis/crdb.cockroachlabs.com/v1alpha1?timeout=32s
{"level":"info","ts":1618866286.0210605,"logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-iris"}
{"level":"info","ts":1618866286.0211103,"logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"sklearn-iris"}
{"level":"info","ts":1618866286.0369768,"logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kubeflow.org/v1beta1","isvc":"sklearn-iris"}
{"level":"info","ts":1618866291.0846086,"logger":"PredictorReconciler","msg":"Reconciling Predictor","PredictorSpec":{"sklearn":{"storageUri":"gs://kfserving-samples/models/sklearn/iris","runtimeVersion":"v0.5.0-rc2","protocolVersion":"v1","name":"kfserving-container","resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}}}
{"level":"info","ts":1618866291.0851047,"logger":"KsvcReconciler","msg":"Creating knative service","namespace":"kfserving-test","name":"sklearn-iris-predictor-default"}
{"level":"info","ts":1618866291.1889343,"logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-iris"}
{"level":"info","ts":1618866291.2164364,"logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kubeflow.org/v1beta1","isvc":"sklearn-iris"}
{"level":"info","ts":1618866291.2165887,"logger":"PredictorReconciler","msg":"Reconciling Predictor","PredictorSpec":{"sklearn":{"storageUri":"gs://kfserving-samples/models/sklearn/iris","runtimeVersion":"v0.5.0-rc2","protocolVersion":"v1","name":"kfserving-container","resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}}}
{"level":"info","ts":1618866291.2169046,"logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-iris"}
{"level":"info","ts":1618866823.958955,"logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"kfserving-test","isvc":{"sklearn":{"storageUri":"gs://seldon-models/sklearn/iris","protocolVersion":"v2","name":"","resources":{}}}}

openshift cluster version: 4.6.22 kfserving version: 0.5.0-rc2

[pvaneck]

Hey @gaodan-fang. I’m glad you got it working. Not sure why the route and istio-ingressgateway node-port didn’t work, but good to see that at least the external-ip worked.

When the current iteration of the guide was made, the sklearn model was deployed using the v2 predict API (which is still WIP), but I believe you deployed the v1 version of sklearn.yaml which would explain the output differences.

I created a PR to smooth out these inconsistencies as well as expand on some areas like curling and troubleshooting. I also plan to update the quick_install.sh to address the issue you saw the nonexistent kfserving_crds.yaml file. Thanks for mentioning that!

[pvaneck]

@gaodan-fang Can you do a oc get po -n knative-serving. I am wondering if the knative-serving pods are up.

I am currently going over the OpenShift install instructions intending on updating it, but ran into issues with permissions for knative related pods. Docs will hopefully be updated soon, but if you are running into CreateContainerErrors, the easiest way to get around the errors are to run: oc adm policy add-scc-to-group anyuid system:serviceaccounts:knative-serving and potentially oc adm policy add-scc-to-group anyuid system:serviceaccounts:kfserving-test. This bug report is likely related which points to the image (gcr.io/distroless/static:nonroot) which Knative images are based on not working out of the box on OpenShift currently. Seems like a patch for the regression was merged just a few days ago, but not sure when the updated runc package will be included in a patch release for the OpenShift Container Platform.