Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
the state of my kservice is "IngressNotConfigured"
See original GitHub issue/kind bug
What steps did you take and what happened: [A clear and concise description of what the bug is.]
I need to deploy kfserving(0.3.0) on the cluster installed by kfctl_istio_dex.v1.0.1.yaml. I know it can’t work with Dex, so I applied kfserving-ingressgateway Deployment as workaround.
But, the READY state of inferenceservice is “false”, and it still has a probing issue
hbseo@kf01:~$ kubectl -nknative-serving get cm config-istio -o yaml
apiVersion: v1
data:
gateway.knative-serving.knative-ingress-gateway: kfserving-ingressgateway.istio-system.svc.cluster.local
local-gateway.knative-serving.cluster-local-gateway: cluster-local-gateway.istio-system.svc.cluster.local
local-gateway.mesh: mesh
reconcileExternalGateway: "false"
kind: ConfigMap
...
...
hbseo@kf01:~$
hbseo@kf01:~$ kubectl get pod -A | grep ingress
istio-system istio-ingressgateway-77f74c944c-k822s 1/1 Running 0 4d5h
istio-system kfserving-ingressgateway-78854bd8cc-wx944 1/1 Running 0 10m
hbseo@kf01:~$
hbseo@kf01:~$ kubectl get service -A | grep ingress
istio-system istio-ingressgateway NodePort 10.102.222.32 <none> 15020:31434/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:30098/TCP,15030:31377/TCP,15031:31150/TCP,15032:32131/TCP,15443:32113/TCP 4d5h
istio-system kfserving-ingressgateway NodePort 10.97.121.65 <none> 15020:31567/TCP,80:32380/TCP,443:32390/TCP,31400:32400/TCP,15011:30655/TCP,8060:30092/TCP,853:31251/TCP,15029:31970/TCP,15030:31269/TCP,15031:31564/TCP,15032:30594/TCP,15443:32364/TCP 10m
hbseo@kf01:~$
hbseo@kf01:~$ kubectl get virtualservice -nhanbae-seo
NAME GATEWAYS HOSTS AGE
mnist-sample-predictor-default [knative-serving/cluster-local-gateway knative-serving/knative-ingress-gateway] [mnist-sample-predictor-default.hanbae-seo mnist-sample-predictor-default.hanbae-seo.example.com mnist-sample-predictor-default.hanbae-seo.svc mnist-sample-predictor-default.hanbae-seo.svc.cluster.local] 4m16s
mnist-sample-predictor-default-mesh [mesh] [mnist-sample-predictor-default.hanbae-seo mnist-sample-predictor-default.hanbae-seo.svc mnist-sample-predictor-default.hanbae-seo.svc.cluster.local] 4m16s
notebook-hanbae-seo-demo [kubeflow/kubeflow-gateway] [*] 4d3h
notebook-hanbae-seo-test [kubeflow/kubeflow-gateway] [*] 3d8h
hbseo@kf01:~$
hbseo@kf01:~$ kubectl get ksvc -nhanbae-seo
NAME URL LATESTCREATED LATESTREADY READY REASON
mnist-sample-predictor-default http://mnist-sample-predictor-default.hanbae-seo.example.com mnist-sample-predictor-default-jk794 mnist-sample-predictor-default-jk794 Unknown IngressNotConfigured
hbseo@kf01:~$
hbseo@kf01:~$ kubectl get revision -nhanbae-seo
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
mnist-sample-predictor-default-jk794 mnist-sample-predictor-default mnist-sample-predictor-default-jk794 1 True
hbseo@kf01:~$
hbseo@kf01:~$ kubectl describe ksvc -nhanbae-seo
Name: mnist-sample-predictor-default
Namespace: hanbae-seo
Labels: <none>
Annotations: serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
serving.knative.dev/lastModifier: system:serviceaccount:kfserving-system:default
API Version: serving.knative.dev/v1
Kind: Service
Metadata:
Creation Timestamp: 2020-04-17T11:46:44Z
Generation: 1
Owner References:
API Version: serving.kubeflow.org/v1alpha2
Block Owner Deletion: true
Controller: true
Kind: InferenceService
Name: mnist-sample
UID: b6a32b1b-e1e0-49fd-8ddc-85f1ffeb7276
Resource Version: 3953183
Self Link: /apis/serving.knative.dev/v1/namespaces/hanbae-seo/services/mnist-sample-predictor-default
UID: 08cdfe9f-cb2f-41cb-8e97-fa40cfb995a1
Spec:
Template:
Metadata:
Annotations:
autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/minScale: 1
autoscaling.knative.dev/target: 1
internal.serving.kubeflow.org/storage-initializer-sourceuri: pvc://hbseo-pvc/saved_model
queue.sidecar.serving.knative.dev/resourcePercentage: 20
sidecar.istio.io/inject: false
Creation Timestamp: <nil>
Labels:
Component: predictor
Endpoint: default
Model: mnist-sample
serving.kubeflow.org/inferenceservice: mnist-sample
Spec:
Container Concurrency: 0
Containers:
Args:
--port=9000
--rest_api_port=8080
--model_name=mnist-sample
--model_base_path=/mnt/models
Command:
/usr/bin/tensorflow_model_server
Image: regi.local:5000/tensorflow/serving:1.14.0
Name: kfserving-container
Readiness Probe:
Success Threshold: 1
Tcp Socket:
Port: 0
Resources:
Limits:
Cpu: 1
Memory: 2Gi
Requests:
Cpu: 1
Memory: 2Gi
Timeout Seconds: 60
Traffic:
Latest Revision: true
Percent: 100
Status:
Address:
URL: http://mnist-sample-predictor-default.hanbae-seo.svc.cluster.local
Conditions:
Last Transition Time: 2020-04-17T11:46:52Z
Status: True
Type: ConfigurationsReady
Last Transition Time: 2020-04-17T11:46:52Z
Message: Ingress has not yet been reconciled.
Reason: IngressNotConfigured
Status: Unknown
Type: Ready
Last Transition Time: 2020-04-17T11:46:52Z
Message: Ingress has not yet been reconciled.
Reason: IngressNotConfigured
Status: Unknown
Type: RoutesReady
Latest Created Revision Name: mnist-sample-predictor-default-jk794
Latest Ready Revision Name: mnist-sample-predictor-default-jk794
Observed Generation: 1
Traffic:
Latest Revision: true
Percent: 100
Revision Name: mnist-sample-predictor-default-jk794
URL: http://mnist-sample-predictor-default.hanbae-seo.example.com
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 5m27s service-controller Created Configuration "mnist-sample-predictor-default"
Normal Created 5m27s service-controller Created Route "mnist-sample-predictor-default"
Warning InternalError 5m27s service-controller failed to reconcile Configuration: Operation cannot be fulfilled on configurations.serving.knative.dev "mnist-sample-predictor-default": the object has been modified; please apply your changes to the latest version and try again
Warning UpdateFailed 5m20s service-controller Failed to update status for Service "mnist-sample-predictor-default": Operation cannot be fulfilled on services.serving.knative.dev "mnist-sample-predictor-default": the object has been modified; please apply your changes to the latest version and try again
Normal Updated 5m19s (x5 over 5m27s) service-controller Updated Service "mnist-sample-predictor-default"
hbseo@kf01:~$
hbseo@kf01:~$ kubectl describe ingresses.networking.internal.knative.dev -nhanbae-seo
Name: mnist-sample-predictor-default
Namespace: hanbae-seo
Labels: serving.knative.dev/route=mnist-sample-predictor-default
serving.knative.dev/routeNamespace=hanbae-seo
serving.knative.dev/service=mnist-sample-predictor-default
Annotations: networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
serving.knative.dev/lastModifier: system:serviceaccount:kfserving-system:default
API Version: networking.internal.knative.dev/v1alpha1
Kind: Ingress
Metadata:
Creation Timestamp: 2020-04-17T11:46:52Z
Generation: 1
Owner References:
API Version: serving.knative.dev/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Route
Name: mnist-sample-predictor-default
UID: 698a5241-4039-4706-919c-8b7b57e46845
Resource Version: 3953175
Self Link: /apis/networking.internal.knative.dev/v1alpha1/namespaces/hanbae-seo/ingresses/mnist-sample-predictor-default
UID: 565d2bb5-1d4d-431a-83c3-2235884578fd
Spec:
Rules:
Hosts:
mnist-sample-predictor-default.hanbae-seo.svc.cluster.local
mnist-sample-predictor-default.hanbae-seo.example.com
Http:
Paths:
Retries:
Attempts: 3
Per Try Timeout: 10m0s
Splits:
Append Headers:
Knative - Serving - Namespace: hanbae-seo
Knative - Serving - Revision: mnist-sample-predictor-default-jk794
Percent: 100
Service Name: mnist-sample-predictor-default-jk794
Service Namespace: hanbae-seo
Service Port: 80
Timeout: 10m0s
Visibility: ExternalIP
Visibility: ExternalIP
Status:
Conditions:
Last Transition Time: 2020-04-17T11:46:52Z
Message: Waiting for VirtualService to be ready
Reason: Uninitialized
Status: Unknown
Type: LoadBalancerReady
Last Transition Time: 2020-04-17T11:46:52Z
Status: True
Type: NetworkConfigured
Last Transition Time: 2020-04-17T11:46:52Z
Message: Waiting for VirtualService to be ready
Reason: Uninitialized
Status: Unknown
Type: Ready
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 5m43s ingress-controller Created VirtualService "mnist-sample-predictor-default-mesh"
Normal Created 5m43s ingress-controller Created VirtualService "mnist-sample-predictor-default"
Normal Updated 5m43s ingress-controller Updated status for Ingress "mnist-sample-predictor-default"
hbseo@kf01:~$
hbseo@kf01:~$ kubectl get servicerole -nhanbae-seo -o yaml
apiVersion: v1
items:
- apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
annotations:
role: admin
user: admin@kubeflow.org
creationTimestamp: "2020-04-13T07:21:01Z"
generation: 1
name: ns-access-istio
namespace: hanbae-seo
ownerReferences:
- apiVersion: kubeflow.org/v1
blockOwnerDeletion: true
controller: true
kind: Profile
name: hanbae-seo
uid: 0043f0d1-2d39-4053-a739-7e961ee3de99
resourceVersion: "32632"
selfLink: /apis/rbac.istio.io/v1alpha1/namespaces/hanbae-seo/serviceroles/ns-access-istio
uid: 99f1b377-103d-433d-87a3-240744feffc4
spec:
rules:
- services:
- '*'
status: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
hbseo@kf01:~$
hbseo@kf01:~$ kubectl get servicerolebinding -nhanbae-seo -o yaml
apiVersion: v1
items:
- apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
creationTimestamp: "2020-04-17T10:48:39Z"
generation: 1
name: knative-network-istio
namespace: hanbae-seo
resourceVersion: "3914205"
selfLink: /apis/rbac.istio.io/v1alpha1/namespaces/hanbae-seo/servicerolebindings/knative-network-istio
uid: 619b5f2d-43fa-46f4-bb48-0a29739d5cee
spec:
roleRef:
kind: ServiceRole
name: ns-access-istio
subjects:
- user: '*'
- apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
annotations:
role: admin
user: admin@kubeflow.org
creationTimestamp: "2020-04-13T07:21:01Z"
generation: 1
name: owner-binding-istio
namespace: hanbae-seo
ownerReferences:
- apiVersion: kubeflow.org/v1
blockOwnerDeletion: true
controller: true
kind: Profile
name: hanbae-seo
uid: 0043f0d1-2d39-4053-a739-7e961ee3de99
resourceVersion: "32635"
selfLink: /apis/rbac.istio.io/v1alpha1/namespaces/hanbae-seo/servicerolebindings/owner-binding-istio
uid: c670d747-2b7a-431b-bc62-597dbe64e1b2
spec:
roleRef:
kind: ServiceRole
name: ns-access-istio
subjects:
- properties:
request.headers[kubeflow-userid]: admin@kubeflow.org
status: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
hbseo@kf01:~$
hbseo@kf01:~$ kubectl -nknative-serving logs -f networking-istio-5b8c5c6cff-nxsbj
{"level":"info","ts":"2020-04-17T11:46:52.817Z","logger":"istiocontroller.ingress-controller","caller":"ingress/ingress.go:160","msg":"Reconciling ingress: &v1alpha1.Ingress{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"mnist-sample-predictor-default\", GenerateName:\"\", Namespace:\"hanbae-seo\", SelfLink:\"/apis/networking.internal.knative.dev/v1alpha1/namespaces/hanbae-seo/ingresses/mnist-sample-predictor-default\", UID:\"565d2bb5-1d4d-431a-83c3-2235884578fd\", ResourceVersion:\"3953166\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63722720812, loc:(*time.Location)(0x29d6020)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"serving.knative.dev/route\":\"mnist-sample-predictor-default\", \"serving.knative.dev/routeNamespace\":\"hanbae-seo\", \"serving.knative.dev/service\":\"mnist-sample-predictor-default\"}, Annotations:map[string]string{\"networking.knative.dev/ingress.class\":\"istio.ingress.networking.knative.dev\", \"serving.knative.dev/creator\":\"system:serviceaccount:kfserving-system:default\", \"serving.knative.dev/lastModifier\":\"system:serviceaccount:kfserving-system:default\"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:\"serving.knative.dev/v1alpha1\", Kind:\"Route\", Name:\"mnist-sample-predictor-default\", UID:\"698a5241-4039-4706-919c-8b7b57e46845\", Controller:(*bool)(0xc000efc228), BlockOwnerDeletion:(*bool)(0xc000efc229)}}, Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1alpha1.IngressSpec{DeprecatedGeneration:0, TLS:[]v1alpha1.IngressTLS(nil), Rules:[]v1alpha1.IngressRule{v1alpha1.IngressRule{Hosts:[]string{\"mnist-sample-predictor-default.hanbae-seo.svc.cluster.local\", \"mnist-sample-predictor-default.hanbae-seo.example.com\"}, Visibility:\"ExternalIP\", HTTP:(*v1alpha1.HTTPIngressRuleValue)(0xc000a89780)}}, Visibility:\"ExternalIP\"}, Status:v1alpha1.IngressStatus{Status:v1.Status{ObservedGeneration:0, Conditions:v1.Conditions{apis.Condition{Type:\"LoadBalancerReady\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbf9e83ab30ba66c8, ext:363604942266633, loc:(*time.Location)(0x29d6020)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"NetworkConfigured\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbf9e83ab30ba51b8, ext:363604942261238, loc:(*time.Location)(0x29d6020)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbf9e83ab30ba4080, ext:363604942256858, loc:(*time.Location)(0x29d6020)}}}, Reason:\"\", Message:\"\"}}}, LoadBalancer:(*v1alpha1.LoadBalancerStatus)(nil), PublicLoadBalancer:(*v1alpha1.LoadBalancerStatus)(nil), PrivateLoadBalancer:(*v1alpha1.LoadBalancerStatus)(nil)}}","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"454ef10b-e3c5-4154-a0ed-c99ce043ea8e","knative.dev/key":"hanbae-seo/mnist-sample-predictor-default"}
{"level":"info","ts":"2020-04-17T11:46:52.817Z","logger":"istiocontroller.ingress-controller","caller":"ingress/ingress.go:169","msg":"Creating/Updating VirtualServices","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"454ef10b-e3c5-4154-a0ed-c99ce043ea8e","knative.dev/key":"hanbae-seo/mnist-sample-predictor-default"}
{"level":"info","ts":"2020-04-17T11:46:52.826Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:258","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"hanbae-seo\", Name:\"mnist-sample-predictor-default\", UID:\"565d2bb5-1d4d-431a-83c3-2235884578fd\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"3953166\", FieldPath:\"\"}): type: 'Normal' reason: 'Created' Created VirtualService \"mnist-sample-predictor-default-mesh\"","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.835Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:258","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"hanbae-seo\", Name:\"mnist-sample-predictor-default\", UID:\"565d2bb5-1d4d-431a-83c3-2235884578fd\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"3953166\", FieldPath:\"\"}): type: 'Normal' reason: 'Created' Created VirtualService \"mnist-sample-predictor-default\"","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.835Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:258","msg":"Queuing probe for http://mnist-sample-predictor-default.hanbae-seo.example.com:80/, IP: 192.168.147.250:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.835Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:258","msg":"Queuing probe for http://mnist-sample-predictor-default.hanbae-seo.svc.cluster.local:80/, IP: 192.168.147.250:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.835Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:258","msg":"Queuing probe for http://mnist-sample-predictor-default.hanbae-seo.svc:80/, IP: 192.168.147.250:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.835Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:258","msg":"Queuing probe for http://mnist-sample-predictor-default.hanbae-seo:80/, IP: 192.168.147.250:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.835Z","logger":"istiocontroller.ingress-controller","caller":"ingress/ingress.go:227","msg":"Ingress successfully synced","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"454ef10b-e3c5-4154-a0ed-c99ce043ea8e","knative.dev/key":"hanbae-seo/mnist-sample-predictor-default"}
{"level":"info","ts":"2020-04-17T11:46:52.840Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://mnist-sample-predictor-default.hanbae-seo.example.com:80/, IP: 192.168.147.250:80 (depth: 3)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.840Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://mnist-sample-predictor-default.hanbae-seo.svc.cluster.local:80/, IP: 192.168.147.250:80 (depth: 2)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.840Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://mnist-sample-predictor-default.hanbae-seo:80/, IP: 192.168.147.250:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-04-17T11:46:52.840Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://mnist-sample-predictor-default.hanbae-seo.svc:80/, IP: 192.168.147.250:80 (depth: 1)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"error","ts":"2020-04-17T11:46:52.841Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://mnist-sample-predictor-default.hanbae-seo.svc.cluster.local:80/ failed, IP: 192.168.147.250:80, ready: false, error: unexpected status code: want [200], got 403 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"error","ts":"2020-04-17T11:46:52.841Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://mnist-sample-predictor-default.hanbae-seo.example.com:80/ failed, IP: 192.168.147.250:80, ready: false, error: unexpected status code: want [200], got 403 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"error","ts":"2020-04-17T11:46:52.841Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://mnist-sample-predictor-default.hanbae-seo.svc:80/ failed, IP: 192.168.147.250:80, ready: false, error: unexpected status code: want [200], got 403 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"error","ts":"2020-04-17T11:46:52.841Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://mnist-sample-predictor-default.hanbae-seo:80/ failed, IP: 192.168.147.250:80, ready: false, error: unexpected status code: want [200], got 403 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
hbseo@kf01:~$
What did you expect to happen:
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
- Istio Version: 1.3
- Knative Version: 0.11.1
- KFServing Version: 0.3.0
- Kubeflow version: kfctl_istio_dex.v1.0.1
- Minikube version:
- Kubernetes version: (use
kubectl version): 1.15.9 - OS (e.g. from
/etc/os-release): ubuntu 18.04
Issue Analytics
- State:
- Created 3 years ago
- Comments:8 (7 by maintainers)
Top Results From Across the Web
the state of my kservice is "IngressNotConfigured" · Issue #786
I know it can't work with Dex, so I applied kfserving-ingressgateway Deployment as workaround. But, the READY state of inferenceservice is " ...
Read more >Serverless Service shows "IngressNotConfigured" error
No translations currently exist. Issue. When trying to create a Serverless Service as per the example in the documentation, this fails and the ......
Read more >Train a model - | notebook.community
Issue: the state of my kservice is "IngressNotConfigured" kind/bug. Issue: Transformer request to Predictor should be Non Blocking.
Read more >Knative Cookbook - IT-Training.pro
The Knative Configuration maintains the desired state of your deployment, pro‐ viding a clean separation of code and configuration using the twelve-factor ...
Read more >Operation cannot be fulfilled on ingresses.networking.internal ...
i upgraded my Knative to the latest version and since then im getting route-controller operation error when trying to create new service.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Totally agreed, we need to have a KFServing debugging guide.
Anyway it is resolved, thanks Dan Sun