Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Failed calling webhook when running random example experiment

See original GitHub issue

Hi there,

my Katib failed to submit the the random algorithm example https://github.com/kubeflow/katib/blob/master/examples/v1alpha3/random-example.yaml:

Internal error occurred: 
failed calling webhook "mutating.experiment.katib.kubeflow.org": 
Post https://katib-controller.kubeflow.svc:443/mutate-experiments?timeout=30s: 
ssh: rejected: connect failed (Connection refused)

The same experiment worked just fine before several days and I didn’t change anything. All deployments kati-controller, katib-db-manager etc. are in healthy states. Any ideas what is going wrong here?

Thanks.

Environment:

Kubeflow version: v1.0
Kubernetes version: (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", 
GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

Issue Analytics

State:
Created 3 years ago
Comments:9 (4 by maintainers)

Top GitHub Comments

2reactions

andreyvelichcommented, Apr 23, 2020

@Felihong Ok, if you have this problem again, let me know.

1reaction

issue-label-bot[bot]commented, Apr 20, 2020

Issue-Label Bot is automatically applying the labels:

Label	Probability
bug	0.61

Please mark this comment with 👍 or 👎 to give our bot feedback! Links: app homepage, dashboard and code for this bot.

Top Results From Across the Web

Dynamic Admission Control | Kubernetes

failurePolicy defines how unrecognized errors and timeout errors from the admission webhook are handled. Allowed values are Ignore or Fail .

Cannot create pod because of MutatingAdmissionWebhook ...

Cannot create pod because of MutatingAdmissionWebhook failure. The MutatingAdmissionWebhook admission plug-in failed to complete mutation in 13 seconds.

Chapter 4. Pipelines OpenShift Container Platform 4.9

Before this update, pipeline runs failed for pipelines containing tasks that emit ... failed calling webhook "validation.webhook.pipeline.tekton.dev": Post ...

Unable to delete a k8s namespace - kubernetes

To fix this, you need to check the validation webhook and fix or delete it. As OP provided the output of endpoints in...

Webhooks - Messenger Platform - Facebook for Developers

If a notification sent to your server fails, we will immediately try a few more times. Your server should handle deduplication in these...