question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Ambassador crashes when it fails to retrieve server APIs

See original GitHub issue

Describe the bug The ambassador pod goes into crash-loop when it fails to connect to Kubernetes API server before eventually coming up.

+ kubewatch --namespace pavan --sync 'python3 /ambassador/post_update.py' --warmup-delay 10s secrets services
AMBASSADOR: diagd running
AMBASSADOR: waiting
PIDS: 30:ambex 31:diagd 49:kubewatch
2019/04/08 23:32:32 unable to retrieve the complete list of server APIs: custom.api.pavan/v1alpha1: the server is currently unable to handle the request
AMBASSADOR: kubewatch exited with status 1
AMBASSADOR: shutting down (1)
NAME                                  READY   STATUS    RESTARTS   AGE
gateway-7d9d88db4f-nhkqn              1/1     Running   4          49m

Ambassador deployment is named gateway in our setup.

To Reproduce Deploy the ambassador and the service responding to the custom Kubernetes APIs together. If the service is not yet up when ambassador starts watching the namespace, it crashes.

Expected behavior Ambassador should come up without crashing/restarts or there should be a way to configure it to not watch a particular service.

Versions

  • Ambassador: 0.53.1
  • Kubernetes environment: Google Kubernetes Engine
  • Version: 1.12.5

Additional context The custom API service is designed to respond to kubectl requests and is not configured with any ambassador related annotations. Note that the same setup works fine with versions < 0.50+.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:7
  • Comments:17 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
yujunzcommented, May 21, 2019

It works as a workaround for me to delete the error apiservice

kubectl delete apiservice v1beta1.admission.certmanager.k8s.io
apiservice.apiregistration.k8s.io "v1beta1.admission.certmanager.k8s.io" deleted

But it would be good to tolerate such error in ambassador.

0reactions
stale[bot]commented, Nov 8, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Ambassador crashes when it fails to retrieve server APIs #1408
Describe the bug The ambassador pod goes into crash-loop when it fails to connect to Kubernetes API server before eventually coming up.
Read more >
Debugging | Edge Stack
First, check the Ambassador Edge Stack Deployment with the following: kubectl get -n ambassador deployments · Check that the “desired” number of Pods...
Read more >
issue in installing ambassador over kubernetes - Server Fault
The Ambassador readiness probe is failing, causing Kubernetes to restart the pod, and putting you into a crash loop.
Read more >
KB2159286 - FIX: The Remote Registry service crashes when ...
Cause. This issue occurs because the SQL Server Performance Counter uses an incorrect method to call the RegQueryValueEx API to access a registry...
Read more >
Failing to see how ambassador pattern enhances modularity ...
To consider my example, lets say that db crashed and restarts, thus get assigned to a different IP. db-ambassador would have to be...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found