Ambassador not working on fresh installed kubernetes
See original GitHub issueDescribe the bug The Ambassador container within ambassador pod does not spawn up properly on a local cluster with kubernetes 1.10.2
To Reproduce Steps to reproduce the behavior: A fresh install of kubernetes 1.10.2 on a local cluster. Follow the guidance to install ambassador-no-rbac.
Expected behavior expect ambassador to spawn up. but it give error code 137. Both liveness probe and Readiness probe failed with getsockopt: connection refused.
ambassador-6c7dd7799b-2mjzx 1/2 CrashLoopBackOff 13 37m
Versions (please complete the following information):
- Ambassador: 0.32.1
- Kubernetes environment: local cluster installed using kubeadm. Configured with calico.
- Version 1.10.2
- Ubuntu 16.04
Additional context Someone @richarddli suggested this is due to a dns problem, but local dns (/etc/resolve.conf) is configured using 8.8.8.8 and 8.8.4.4, I am sure at least I can access google website. (By deploying a busybox into my cluster, I checked my busybox container can ping google.com, but not sure for others.) Here are some extra log for debugging
kubernetes@local-cluster-0:~/my-kubeflow$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
I0503 10:46:23.833551 1 dns.go:48] version: 1.14.8
I0503 10:46:23.862220 1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0503 10:46:23.862374 1 server.go:119] FLAG: --alsologtostderr="false"
I0503 10:46:23.862433 1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0503 10:46:23.862466 1 server.go:119] FLAG: --config-map=""
I0503 10:46:23.862485 1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0503 10:46:23.862504 1 server.go:119] FLAG: --config-period="10s"
I0503 10:46:23.862532 1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0503 10:46:23.862551 1 server.go:119] FLAG: --dns-port="10053"
I0503 10:46:23.862592 1 server.go:119] FLAG: --domain="cluster.local."
I0503 10:46:23.862621 1 server.go:119] FLAG: --federations=""
I0503 10:46:23.862648 1 server.go:119] FLAG: --healthz-port="8081"
I0503 10:46:23.862668 1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0503 10:46:23.862688 1 server.go:119] FLAG: --kube-master-url=""
I0503 10:46:23.862742 1 server.go:119] FLAG: --kubecfg-file=""
I0503 10:46:23.862761 1 server.go:119] FLAG: --log-backtrace-at=":0"
I0503 10:46:23.862794 1 server.go:119] FLAG: --log-dir=""
I0503 10:46:23.862815 1 server.go:119] FLAG: --log-flush-frequency="5s"
I0503 10:46:23.862835 1 server.go:119] FLAG: --logtostderr="true"
I0503 10:46:23.862854 1 server.go:119] FLAG: --nameservers=""
I0503 10:46:23.862872 1 server.go:119] FLAG: --stderrthreshold="2"
I0503 10:46:23.862891 1 server.go:119] FLAG: --v="2"
I0503 10:46:23.862911 1 server.go:119] FLAG: --version="false"
I0503 10:46:23.862953 1 server.go:119] FLAG: --vmodule=""
I0503 10:46:23.863269 1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0503 10:46:23.864231 1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0503 10:46:23.864282 1 dns.go:146] Starting endpointsController
I0503 10:46:23.864363 1 dns.go:149] Starting serviceController
I0503 10:46:23.901849 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0503 10:46:23.901936 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0503 10:46:24.364803 1 dns.go:170] Initialized services and endpoints from apiserver
I0503 10:46:24.364871 1 server.go:135] Setting up Healthz Handler (/readiness)
I0503 10:46:24.364935 1 server.go:140] Setting up cache handler (/cache)
I0503 10:46:24.364962 1 server.go:126] Status HTTP port 8081
I0504 04:33:43.169511 1 dns.go:555] Could not find endpoints for service "tf-hub-0" in namespace "kubeflow". DNS records will be created once endpoints show up.
I0515 01:02:23.572540 1 dns.go:555] Could not find endpoints for service "tf-hub-0" in namespace "kubeflow". DNS records will be created once endpoints show up.
kubernetes@local-cluster-0:~/my-kubeflow$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
I0503 10:46:24.860136 1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0503 10:46:24.870496 1 nanny.go:94] Starting dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0503 10:46:25.334307 1 nanny.go:116] dnsmasq[12]: started, version 2.78 cachesize 1000
I0503 10:46:25.334618 1 nanny.go:116] dnsmasq[12]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0503 10:46:25.334632 1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0503 10:46:25.334638 1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0503 10:46:25.334643 1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0503 10:46:25.334651 1 nanny.go:116] dnsmasq[12]: reading /etc/resolv.conf
I0503 10:46:25.334657 1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0503 10:46:25.334662 1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0503 10:46:25.334668 1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0503 10:46:25.334672 1 nanny.go:116] dnsmasq[12]: using nameserver 8.8.8.8#53
I0503 10:46:25.334678 1 nanny.go:116] dnsmasq[12]: using nameserver 8.8.4.4#53
I0503 10:46:25.334716 1 nanny.go:116] dnsmasq[12]: read /etc/hosts - 7 addresses
I0503 10:46:25.334851 1 nanny.go:119]
W0503 10:46:25.334861 1 nanny.go:120] Got EOF from stdout
kubernetes@local-cluster-0:~/my-kubeflow$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
I0503 10:46:25.240827 1 main.go:51] Version v1.14.8
I0503 10:46:25.240868 1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
I0503 10:46:25.240906 1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}
I0503 10:46:25.240992 1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}
W0503 10:46:25.241374 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:58432->127.0.0.1:53: read: connection refused
Can anyone help me have a look? Many thanks in advance!
Issue Analytics
- State:
- Created 5 years ago
- Comments:41 (21 by maintainers)
Top GitHub Comments
@victortrac can you please provide more detail how you fixed this?
OK, so here’s a better Ambassador deployment YAML for Seldon. I’ll figure out how to create a PR for this for the Seldon folks.