Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

x509: certificate signed by unknown authority while running sample example

See original GitHub issue

What steps did you take and what happened: i installed kfserving and all the pods are up .

 kubectl get pods --namespace kfserving-system
NAME                             READY   STATUS    RESTARTS   AGE
kfserving-controller-manager-0   2/2     Running   1          12h

all the pods are up and running in the knative-serving namespace as well.

k get pods -n knative-serving
NAME                               READY   STATUS    RESTARTS   AGE
activator-76984478f7-vbsh9         1/1     Running   0          21h
autoscaler-598d974c99-46pmq        1/1     Running   1          21h
controller-9b998cd47-4l255         1/1     Running   0          21h
istio-webhook-69cd874949-hq9k2     1/1     Running   0          21h
networking-istio-df55795c6-hznj5   1/1     Running   0          21h
webhook-658874f97-bxjl5            1/1     Running   1          21h

but while running the examples from the developer guide developer_guide get this error :

kubectl apply -f docs/samples/tensorflow/tensorflow.yaml
Error from server (InternalError): error when creating "docs/samples/tensorflow/tensorflow.yaml": Internal error occurred: failed calling webhook "inferenceservice.kfserving-webhook-server.defaulter": Post https://kfserving-webhook-server-service.kfserving-system.svc:443/mutate-inferenceservices?timeout=30s: x509: certificate signed by unknown authority

What did you expect to happen:

examples run successfully and see model serving deployment running under default the specified namespace.

Anything else you would like to add:

i tried finding similar issues on the web but could not find exact same issue being reported.

certificate manager is installed in accordance with documentation step in the developer guide : developer_guide_use_certmanager

installation is done using manifest file :

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.16.0/cert-manager.yaml

cert-manager pods are up and running :

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-69779b98cd-b85f5              1/1     Running   0          11h
cert-manager-cainjector-7c4c4bbbb9-rqzfz   1/1     Running   1          11h
cert-manager-webhook-6496b996cb-4fcxg      1/1     Running   0          11h
i318056@C02XX0SEJGH5 kfserving %

logs from make deploy. :

make deploy
/Users/i318056/go/bin/controller-gen "crd:maxDescLen=0" paths=./pkg/apis/... output:crd:dir=config/crd
/Users/i318056/go/bin/controller-gen rbac:roleName=kfserving-manager-role paths=./pkg/controller/inferenceservice/... output:rbac:artifacts:config=config/rbac
/Users/i318056/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths=./pkg/apis/serving/v1alpha2
#TODO Remove this until new controller-tools is released
perl -pi -e 's/storedVersions: null/storedVersions: []/g' config/crd/serving.kubeflow.org_inferenceservices.yaml
perl -pi -e 's/conditions: null/conditions: []/g' config/crd/serving.kubeflow.org_inferenceservices.yaml
perl -pi -e 's/Any/string/g' config/crd/serving.kubeflow.org_inferenceservices.yaml
# Remove the certmanager certificate if KFSERVING_ENABLE_SELF_SIGNED_CA is not false
cd config/default && if [ true != false ]; then \
kustomize edit remove resource certmanager/certificate.yaml; \
else kustomize edit add resource certmanager/certificate.yaml; fi;
kustomize build config/default | kubectl apply --validate=false -f -
namespace/kfserving-system unchanged
customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kubeflow.org configured
role.rbac.authorization.k8s.io/leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/kfserving-manager-role configured
clusterrole.rbac.authorization.k8s.io/kfserving-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kfserving-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kfserving-proxy-rolebinding unchanged
configmap/inferenceservice-config unchanged
secret/kfserving-webhook-server-secret unchanged
service/kfserving-controller-manager-metrics-service unchanged
service/kfserving-controller-manager-service unchanged
service/kfserving-webhook-server-service unchanged
statefulset.apps/kfserving-controller-manager unchanged
certificate.cert-manager.io/serving-cert created
issuer.cert-manager.io/selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured
validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured
if [ true != false ]; then ./hack/self-signed-ca.sh; fi;
service: kfserving-webhook-server-service
namespace: kfserving-system
secret: kfserving-webhook-server-cert
webhookDeploymentName: kfserving-controller-manager-0
webhookConfigName: inferenceservice.serving.kubeflow.org
creating certs in tmpdir /var/folders/tq/pj7bbsk55495fn0fkv17zxb80000gn/T/tmp.G03Grhq6
Generating RSA private key, 2048 bit long modulus
.................................................................................................+++
...................................+++
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
..............+++
........+++
e is 65537 (0x10001)
Signature ok
subject=/CN=kfserving-webhook-server-service.kfserving-system.svc
Getting CA Private Key
secret/kfserving-webhook-server-cert configured
pod "kfserving-controller-manager-0" deleted
webhook kfserving-controller-manager-0 is restarted to utilize the new secret
CA Certificate:
-----BEGIN CERTIFICATE-----
MIIC/DCCAeQCCQCErGeGtw6WXDANBgkqhkiG9w0BAQsFADBAMT4wPAYDVQQDDDVr
ZnNlcnZpbmctd2ViaG9vay1zZXJ2ZXItc2VydmljZS5rZnNlcnZpbmctc3lzdGVt
LnN2YzAeFw0yMDA3MzAwNTIwMjNaFw0yMTA3MzAwNTIwMjNaMEAxPjA8BgNVBAMM
NWtmc2VydmluZy13ZWJob29rLXNlcnZlci1zZXJ2aWNlLmtmc2VydmluZy1zeXN0
ZW0uc3ZjMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv4aPGXduSeUr
VfnH5HztEgJYhMthgXwZEMCXw2MFXsslA+TrFi4jV7d9k4ZbBZwSO7vKvqiJk8ob
n2jObKq6+9+18xIOuz4u0mtb4VHvgUgElsUtukWk61dxKEvKrRJ53UbZqB7y+8CY
pMA1IHwSiZfPYMVf2vIBvb3qf7HiGtdYK8c/tw0GHuXjCnDYiRwPhlYlb27qkOb/
4hKrcFvyx2CkeAahbSXOJhrKZIrKKRIt3BVh/KiuKfKLGDUyipEzp+Xom8gs8FOM
/FDQbhcPwaI9QhSHoblRSNA7FHpnKo72jZfbL0KFiyRxKw6oRTroKK/JFId01PED
d5Wp3iTzCwIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBn7YghbNYB7jBy4xF2fuBp
rwtGl1VJlyOFTFQga+QWw4BQBf9+A23fJDGJk0kOqHiSiXM0ehwE34LZtkz2cv7y
K2+gmBLHzhc2ssM7+V1DuiCBHk/iOOa4Q6cvISVNgaaUOoAHfO9bc8DENqkDLkH6
ioinCUX4ddW5m2+nqn9f/ydcwrlCobytQQhSQwcqH5WAaKRrYuKAPhmmQnoPzISw
hSh2sajEN+5GRfGkwAkNsBPQ/CQ3YPpKI0mI296KKYJ10feA1xPpsw5djrmRn2wL
96e72Q1lvL3PKn3SBjZokHO4y3aIbUaVrUtZuhXgDTBIbwcLU1MPxVjCosnVRln7
-----END CERTIFICATE-----
Encoded CA:
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvRENDQWVRQ0NRQ0VyR2VHdHc2V1hEQU5CZ2txaGtpRzl3MEJBUXNGQURCQU1UNHdQQVlEVlFRREREVnIKWm5ObGNuWnBibWN0ZDJWaWFHOXZheTF6WlhKMlpYSXRjMlZ5ZG1salpTNXJabk5sY25acGJtY3RjM2x6ZEdWdApMbk4yWXpBZUZ3MHlNREEzTXpBd05USXdNak5hRncweU1UQTNNekF3TlRJd01qTmFNRUF4UGpBOEJnTlZCQU1NCk5XdG1jMlZ5ZG1sdVp5MTNaV0pvYjI5ckxYTmxjblpsY2kxelpYSjJhV05sTG10bWMyVnlkbWx1WnkxemVYTjAKWlcwdWMzWmpNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXY0YVBHWGR1U2VVcgpWZm5INUh6dEVnSlloTXRoZ1h3WkVNQ1h3Mk1GWHNzbEErVHJGaTRqVjdkOWs0WmJCWndTTzd2S3ZxaUprOG9iCm4yak9iS3E2KzkrMTh4SU91ejR1MG10YjRWSHZnVWdFbHNVdHVrV2s2MWR4S0V2S3JSSjUzVWJacUI3eSs4Q1kKcE1BMUlId1NpWmZQWU1WZjJ2SUJ2YjNxZjdIaUd0ZFlLOGMvdHcwR0h1WGpDbkRZaVJ3UGhsWWxiMjdxa09iLwo0aEtyY0Z2eXgyQ2tlQWFoYlNYT0pocktaSXJLS1JJdDNCVmgvS2l1S2ZLTEdEVXlpcEV6cCtYb204Z3M4Rk9NCi9GRFFiaGNQd2FJOVFoU0hvYmxSU05BN0ZIcG5LbzcyalpmYkwwS0ZpeVJ4S3c2b1JUcm9LSy9KRklkMDFQRUQKZDVXcDNpVHpDd0lEQVFBQk1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQm43WWdoYk5ZQjdqQnk0eEYyZnVCcApyd3RHbDFWSmx5T0ZURlFnYStRV3c0QlFCZjkrQTIzZkpER0prMGtPcUhpU2lYTTBlaHdFMzRMWnRrejJjdjd5CksyK2dtQkxIemhjMnNzTTcrVjFEdWlDQkhrL2lPT2E0UTZjdklTVk5nYWFVT29BSGZPOWJjOERFTnFrRExrSDYKaW9pbkNVWDRkZFc1bTIrbnFuOWYveWRjd3JsQ29ieXRRUWhTUXdjcUg1V0FhS1JyWXVLQVBobW1Rbm9QeklTdwpoU2gyc2FqRU4rNUdSZkdrd0FrTnNCUFEvQ1EzWVBwS0kwbUkyOTZLS1lKMTBmZUExeFBwc3c1ZGpybVJuMndMCjk2ZTcyUTFsdkwzUEtuM1NCalpva0hPNHkzYUliVWFWclV0WnVoWGdEVEJJYndjTFUxTVB4VmpDb3NuVlJsbjcKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

patching ca bundle for mutating webhook configuration...
mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org patched
patching ca bundle for validating webhook configuration...
validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org patched

Environment:

Istio Version: 1.6.5
Knative Version: 0.16.0
KFServing Version: fork from latest master.
Kubeflow version: fork from latest master.
Minikube version: using Docker for Desktop. version : 2.3.0.3
Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}