ksvc Uninitialized with Waiting for load balancer to be ready error

/kind bug

What steps did you take and what happened: I deployed the following custom predictor inference service definition:

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  annotations:
    autoscaling.knative.dev/maxScale: '5'
    autoscaling.knative.dev/minScale: '1'
  labels:
    app: iris
    app.kubernetes.io/instance: iris-dev
    controller-tools.k8s.io: '1.0'
    env: mlk8s-dev
    sidecar.istio.io/inject: 'false'
    version: 0.0.35
  name: iris
  namespace: dev
spec:
  default:
    predictor:
      custom:
        container:
          env:
            - name: PATH_TO_MODEL
              value: '/mnt/models/iris/models/2021-05-05T18:47:18.359991/'
            - name: STORAGE_URI
              value: 'pvc://dev-pvc'
            - name: MODEL_VERSION
              value: 0.0.35
          image: 'custom-image/iris-kfserving:0.0.35'
          name: kfserving-container
          resources:
            limits:
              cpu: '1'
              memory: 6Gi
            requests:
              cpu: 250m
              memory: 4Gi

What did you expect to happen: The inference service to be in READY status True and routes to be setup correctly.

Anything else you would like to add: I installed kubeflow through aws argoflow distribution . I noticed there is a missing virtual service:

kubectl get vs -n dev | grep iris
iris-predictor-default-ingress                                ["istio-system/cluster-local-gateway","kubeflow/kubeflow-gateway"]   ["iris-predictor-default.dev","iris-predictor-default.dev.serving.domain-name","iris-predictor-default.dev.svc","iris-predictor-default.dev.svc.cluster.local"]                                                                                                                       22m
iris-predictor-default-mesh                                   ["mesh"]                                                             ["iris-predictor-default.dev","iris-predictor-default.dev.svc","iris-predictor-default.dev.svc.cluster.local"]                                                                                                                                                                             22m

Environment:

• Istio Version: 1.10.2
• Knative Version: v0.24.0
• KFServing Version: v0.6.0
• Kubeflow version: v1.3.0
• Kfdef: N/A
• Minikube version: EKS cluster
• Kubernetes version: (use kubectl version): kubectl version Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.4-dirty", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"dirty", BuildDate:"2021-03-15T10:03:32Z", GoVersion:"go1.16.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
• OS (e.g. from /etc/os-release):

[amybachir]

@edwardzjl what does your config-istio configmap looks like? What is the value of enable-virtualservice-status ?

Can you run the following command: kubectl describe cm config-istio -n knative-serving

[markwinter]

We started having this issue after having a problem with istio getting OOM failures. After fixing istio I could only fix the ksvc “Waiting for load balancer” issue by forcing a new revision update.

We also recently changed the expiry time on the kserve certificate, perhaps also related