KFserving explanations fail when used with authenticated kubeflow

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

I am using kfserving deployments in a kubeflow cluster with an authentication specified by the arrikto config. https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_existing_arrikto.yaml. When I make explanation requests with relevant headers(for auth) I get a 404 response, but the prediction works. I am using this sample from the docs https://github.com/kubeflow/kfserving/tree/master/docs/samples/explanation/alibi/income After some debugging, we found that the explainer is unable to get an output from the predictors as the istio-ingress gateway redirects these requests to a Dex URL for login. (Even the requests internal to the cluster).

What did you expect to happen: The explainer should have responded with the relevant explanation.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] The service type for the default predictor is ExternalName instead of ClusterIP.

Environment:

• Istio Version:
• Knative Version:
• KFServing Version:
• Kubeflow version:
• Minikube version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[yuzisun]

Please check https://github.com/kubeflow/kfserving/blob/master/docs/samples/istio-dex/README.md

/close

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.