503 Service Unavailable" when running custom sample in kubeflow kfserving

/kind bug

What steps did you take and what happened: I created the inferenceservice in the kubeflow with the sample , inferenceservice is READY, but the predict api did not return the expect response message ‘Hello Python KFServing Sample!’

What did you expect to happen: Get the response message with code 200 not 503.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] Here is my inferenceservice:

NAME            URL                                                                       READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
custom-sample   http://custom-sample.kfserving-test.example.com/v1/models/custom-sample   True    100                                20m

Here is my test request, it got 503 error. curl -v -H “Host:custom-sample.kfserving-test.example.com” http://my_host:my_port/v1/models/custom-sample:predict

* Expire in 0 ms for 6 (transfer 0x1f1cfd09100)
* Expire in 1 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 1 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 2 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 7 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 9 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 4 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 50 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 50 ms for 1 (transfer 0x1f1cfd09100)
* Expire in 50 ms for 1 (transfer 0x1f1cfd09100)
*   Trying 10.12.202.49...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x1f1cfd09100)
* Connected to my_host port my_port (#0)
> GET /v1/models/custom-sample:predict HTTP/1.1
> Host:custom-sample.kfserving-test.example.com
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< date: Wed, 01 Jul 2020 09:44:07 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host my_host left intact

here is my virtualservice:

NAME                                   GATEWAYS                                                            HOSTS                                                                                                                                                                                                                             AGE
custom-sample                          [kubeflow-gateway.kubeflow]                                         [custom-sample.kfserving-test.example.com]                                                                                                                                                                                        6m56s
custom-sample-predictor-default        [knative-serving/cluster-local-gateway kubeflow/kubeflow-gateway]   [custom-sample-predictor-default.kfserving-test custom-sample-predictor-default.kfserving-test.example.com custom-sample-predictor-default.kfserving-test.svc custom-sample-predictor-default.kfserving-test.svc.cluster.local]   6m57s
custom-sample-predictor-default-mesh   [mesh]                                                              [custom-sample-predictor-default.kfserving-test custom-sample-predictor-default.kfserving-test.svc custom-sample-predictor-default.kfserving-test.svc.cluster.local]                                                              6m57s

Environment:

• Istio Version: 1.4.5
• Knative Version: 1.0.0
• KFServing Version: 1.0.0
• Kubeflow version:
• Kfdef: k8s_istio
• Minikube version:
• Kubernetes version: v1.17.4
• OS (e.g. from /etc/os-release):

[ValdasK]

I have exactly same issue, all samples that I try to install give 503 response. I’m also using GCP IAP and your @yuzisun examples on how to set them up.

[connorlwilkes]

@yuzisun I am facing this issue also. I am able to curl the service (sklearn-iris-predictor-default.namespace.svc.cluster.local) completely fine from within the cluster but when I run from outside the cluster I am getting ‘503 errors’. I have turned on access logging and I can see the following in my cluster-local-gateway:

{"duration":"0","downstream_local_address":"10.1.0.198:80","upstream_transport_failure_reason":"-","response_code":"503","user_agent":"python-requests/2.24.0","response_flags":"NR","start_time":"2020-10-29T21:53:14.888Z","method":"GET","request_id":"0464fb51-04b4-4517-9f37-728977f2f779","upstream_host":"-","x_forwarded_for":"194.62.232.110, 34.120.137.230,10.10.10.6,10.1.2.2","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"-","downstream_remote_address":"10.1.2.2:41212","path":"/kfserving/namespace/sklearn-iris:predict","authority":"sklearn-iris-predictor-default.namespace.svc.cluster.local","protocol":"HTTP/2","upstream_service_time":"-","upstream_local_address":"-"}

and in my ingress gateway:

{"authority":"sklearn-iris-predictor-default.namespace.svc.cluster.local","path":"/kfserving/namespace/sklearn-iris:predict","protocol":"HTTP/1.1","upstream_service_time":"71","upstream_local_address":"10.1.2.2:55614","duration":"72","upstream_transport_failure_reason":"-","route_name":"-","downstream_local_address":"10.1.2.2:80","user_agent":"python-requests/2.24.0","response_code":"503","response_flags":"URX","start_time":"2020-10-29T21:53:14.886Z","method":"GET","request_id":"35287f1e-cc46-46a4-8b6d-b49249cf9a5f","upstream_host":"10.1.2.215:80","x_forwarded_for":"194.62.232.110, 34.120.137.230,10.10.10.6","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|80||cluster-local-gateway.istio-system.svc.cluster.local","downstream_remote_address":"10.10.10.6:55906"}