Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Exception: Bad response from application: 503 when sending a prediction request through ingress and IAP for KFServing GCP/IAP Example

See original GitHub issue

/kind bug

What steps did you take and what happened: When trying KFServing GCP/IAP example from this page https://github.com/kubeflow/kfserving/tree/master/docs/samples/gcp-iap, I get a 404 error when testing the external predict endpoint using iap_request.py

Just for information, when sending requests, I am using the service account which was created when deploying Kubeflow 1.1 using gcp-blueprints (name of this service account is ${KF_NAME}-user) I am note sure to understand the note at the end of the page : Making requests to the service may be blocked (403) by the new istio sidecar container until a new AuthorizationPolicy is added that allows access to this inference URI from a specified ServiceAccount or Namespace.

I am not sure this is the cause of the 404 error, but Please could you give some clarification about this note ? What kind of AuthorizationPolicy must be added and how ?

And how could I solve the 404 error below ?

What did you expect to happen: Return code 200

Anything else you would like to add: Here are the commands I executed

export PROJECT=xxxx
gcloud config set project ${PROJECT}
export ZONE=yyyyyy
gcloud config set compute/zone ${ZONE}

export KF_NAME=zzzzz
kubens kfnamespace

cd ~/repos/kubeflow/kfserving/docs/samples/gcp-iap/

kubectl apply -f sklearn-iap-with-authz.yaml
kubectl apply -f virtual-service.yaml

I check that the pod is successfully running

kubectl get po
NAME                                                              READY   STATUS      RESTARTS   AGE
sklearn-iap-predictor-default-wl4zb-deployment-56fbfc47f4-m2vcr   3/3     Running     0          25s

I set the service account as the active account

export SERVICE_ACCOUNT=my_sa_name
export KEY_FILE=${HOME}/.config/gcloud/${SERVICE_ACCOUNT}-credentials.json
gcloud config set account ${SERVICE_ACCOUNT}@${PROJECT}.iam.gserviceaccount.com
export GOOGLE_APPLICATION_CREDENTIALS=${KEY_FILE}

I get the value for INGRESS_DNS (which must be updated in make-prediction.sh) kubectl -n istio-system get ingress

I update the variables in make-prediction.sh vi make-prediction.sh

INFERENCE_SERVICE=sklearn-iap
INPUT_PATH=@./iris-input.json
PROJECT='xxxx'
NAMESPACE='kfnamespace'
INGRESS_DNS=${KF_NAME}.endpoints.${PROJECT}.cloud.goog
export IAP_CLIENT_ID='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.apps.googleusercontent.com'
SERVICE_URL=https://${INGRESS_DNS}/kfserving/${NAMESPACE}/${INFERENCE_SERVICE}:predict

I send the query python iap_request.py https://zzzzz.endpoints.xxxx.cloud.goog/kfserving/kfnamespace/sklearn-iap:predict aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.apps.googleusercontent.com --input=./iris-input.json

I get the response below (error 404)

~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:979: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zzzzz.endpoints.xxxx.cloud.goog'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
Traceback (most recent call last):
  File "iap_request.py", line 143, in <module>
    main()
  File "iap_request.py", line 136, in main
    raise Exception('Bad response from application: {!r} / {!r} / {!r}'.format(
Exception: Bad response from application: 404 / {'x-powered-by': 'Express', 'content-security-policy': "default-src 'none'", 'x-content-type-options': 'nosniff', 'content-type': 'text/html; charset=utf-8', 'content-length': '173', 'date': 'Wed, 11 Nov 2020 08:53:44 GMT', 'x-envoy-upstream-service-time': '1', 'server': 'istio-envoy', 'Via': '1.1 google', 'Alt-Svc': 'clear'} / '<!DOCTYPE html>\n<html lang="en">\n<head>\n<meta charset="utf-8">\n<title>Error</title>\n</head>\n<body>\n<pre>Cannot POST /kfserving/kfnamespace/sklearn-iap:predict</pre>\n</body>\n</html>\n'

I checked that ${GOOGLE_APPLICATION_CREDENTIALS} points to the key json file, and it is OK more ${GOOGLE_APPLICATION_CREDENTIALS}

I checked that the active account is the service account : gcloud auth list

Environment:

Istio Version:
Knative Version:
KFServing Version:
Kubeflow version: build version v1beta1 (Kubeflow 1.1 deployed in GCP using gcp-blueprints)
Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm] : gcp-blueprints
Minikube version: GCP
Kubernetes version: (use kubectl version): Major:“1”, Minor:“17+”, GitVersion:"v1.17.12-gke.1504
OS (e.g. from /etc/os-release):

Issue Analytics

State:
Created 3 years ago
Comments:31 (22 by maintainers)

Top GitHub Comments

1reaction

jal06commented, Nov 30, 2020

@Bobgy,

Thanks for the link. FYI, I opened an issue for the connection to kfserving endpoint in kubeflow/gcp-blueprints, and I also opened a PR for adding the deployment of the local geteway in the namespace istio-system, which from my understanding, solves the issue.

I will also read accurately the documentation you gave and try the connexion via curl. After that, I will create another issue about authentication via IAP in kubeflow/gcp-blueprints. Hopefully, it will work, and in this case, it should be useful to add some documentation in Kubeflow (maybe just pointing to this documentation); and if I still have problemns with authentication via IAP, I will describe them.

1reaction

yuzisuncommented, Nov 19, 2020

@jal06 Can you check the kfdef yaml? the istio local gateway needs to be defined e.g here . Also do not get confused with Istio gateway resource and Istio ingress gateway deployment, gateway resource cluster-local-gateway in knative-serving namespace defines the listeners for the Istio local gateway deployment, the actual istio local gateway deployment runs in istio-system.