question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Ambassador not working on fresh installed kubernetes

See original GitHub issue

Describe the bug The Ambassador container within ambassador pod does not spawn up properly on a local cluster with kubernetes 1.10.2

To Reproduce Steps to reproduce the behavior: A fresh install of kubernetes 1.10.2 on a local cluster. Follow the guidance to install ambassador-no-rbac.

Expected behavior expect ambassador to spawn up. but it give error code 137. Both liveness probe and Readiness probe failed with getsockopt: connection refused.

ambassador-6c7dd7799b-2mjzx   1/2       CrashLoopBackOff   13         37m

Versions (please complete the following information):

  • Ambassador: 0.32.1
  • Kubernetes environment: local cluster installed using kubeadm. Configured with calico.
  • Version 1.10.2
  • Ubuntu 16.04

Additional context Someone @richarddli suggested this is due to a dns problem, but local dns (/etc/resolve.conf) is configured using 8.8.8.8 and 8.8.4.4, I am sure at least I can access google website. (By deploying a busybox into my cluster, I checked my busybox container can ping google.com, but not sure for others.) Here are some extra log for debugging

kubernetes@local-cluster-0:~/my-kubeflow$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
I0503 10:46:23.833551       1 dns.go:48] version: 1.14.8
I0503 10:46:23.862220       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0503 10:46:23.862374       1 server.go:119] FLAG: --alsologtostderr="false"
I0503 10:46:23.862433       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0503 10:46:23.862466       1 server.go:119] FLAG: --config-map=""
I0503 10:46:23.862485       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0503 10:46:23.862504       1 server.go:119] FLAG: --config-period="10s"
I0503 10:46:23.862532       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0503 10:46:23.862551       1 server.go:119] FLAG: --dns-port="10053"
I0503 10:46:23.862592       1 server.go:119] FLAG: --domain="cluster.local."
I0503 10:46:23.862621       1 server.go:119] FLAG: --federations=""
I0503 10:46:23.862648       1 server.go:119] FLAG: --healthz-port="8081"
I0503 10:46:23.862668       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0503 10:46:23.862688       1 server.go:119] FLAG: --kube-master-url=""
I0503 10:46:23.862742       1 server.go:119] FLAG: --kubecfg-file=""
I0503 10:46:23.862761       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0503 10:46:23.862794       1 server.go:119] FLAG: --log-dir=""
I0503 10:46:23.862815       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0503 10:46:23.862835       1 server.go:119] FLAG: --logtostderr="true"
I0503 10:46:23.862854       1 server.go:119] FLAG: --nameservers=""
I0503 10:46:23.862872       1 server.go:119] FLAG: --stderrthreshold="2"
I0503 10:46:23.862891       1 server.go:119] FLAG: --v="2"
I0503 10:46:23.862911       1 server.go:119] FLAG: --version="false"
I0503 10:46:23.862953       1 server.go:119] FLAG: --vmodule=""
I0503 10:46:23.863269       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0503 10:46:23.864231       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0503 10:46:23.864282       1 dns.go:146] Starting endpointsController
I0503 10:46:23.864363       1 dns.go:149] Starting serviceController
I0503 10:46:23.901849       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0503 10:46:23.901936       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0503 10:46:24.364803       1 dns.go:170] Initialized services and endpoints from apiserver
I0503 10:46:24.364871       1 server.go:135] Setting up Healthz Handler (/readiness)
I0503 10:46:24.364935       1 server.go:140] Setting up cache handler (/cache)
I0503 10:46:24.364962       1 server.go:126] Status HTTP port 8081
I0504 04:33:43.169511       1 dns.go:555] Could not find endpoints for service "tf-hub-0" in namespace "kubeflow". DNS records will be created once endpoints show up.
I0515 01:02:23.572540       1 dns.go:555] Could not find endpoints for service "tf-hub-0" in namespace "kubeflow". DNS records will be created once endpoints show up.
kubernetes@local-cluster-0:~/my-kubeflow$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
I0503 10:46:24.860136       1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0503 10:46:24.870496       1 nanny.go:94] Starting dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0503 10:46:25.334307       1 nanny.go:116] dnsmasq[12]: started, version 2.78 cachesize 1000
I0503 10:46:25.334618       1 nanny.go:116] dnsmasq[12]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0503 10:46:25.334632       1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0503 10:46:25.334638       1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0503 10:46:25.334643       1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0503 10:46:25.334651       1 nanny.go:116] dnsmasq[12]: reading /etc/resolv.conf
I0503 10:46:25.334657       1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0503 10:46:25.334662       1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0503 10:46:25.334668       1 nanny.go:116] dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0503 10:46:25.334672       1 nanny.go:116] dnsmasq[12]: using nameserver 8.8.8.8#53
I0503 10:46:25.334678       1 nanny.go:116] dnsmasq[12]: using nameserver 8.8.4.4#53
I0503 10:46:25.334716       1 nanny.go:116] dnsmasq[12]: read /etc/hosts - 7 addresses
I0503 10:46:25.334851       1 nanny.go:119]
W0503 10:46:25.334861       1 nanny.go:120] Got EOF from stdout
kubernetes@local-cluster-0:~/my-kubeflow$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
I0503 10:46:25.240827       1 main.go:51] Version v1.14.8
I0503 10:46:25.240868       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
I0503 10:46:25.240906       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}
I0503 10:46:25.240992       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}
W0503 10:46:25.241374       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:58432->127.0.0.1:53: read: connection refused

Can anyone help me have a look? Many thanks in advance!

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:41 (21 by maintainers)

github_iconTop GitHub Comments

3reactions
ttfreemancommented, Sep 18, 2019

@victortrac can you please provide more detail how you fixed this?

1reaction
kflynncommented, Jun 8, 2018

OK, so here’s a better Ambassador deployment YAML for Seldon. I’ll figure out how to create a PR for this for the Seldon folks.

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ambassador
rules:
- apiGroups: [""]
  resources:
  - services
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["create", "update", "patch", "get", "list", "watch"]
- apiGroups: [""]
  resources:
  - secrets
  verbs: ["get", "list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ambassador
  namespace: seldon
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ambassador
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ambassador
subjects:
- kind: ServiceAccount
  name: ambassador
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ambassador
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ambassador
subjects:
- kind: ServiceAccount
  name: ambassador
  namespace: seldon
---
apiVersion: v1
kind: Service
metadata:
  name: ambassador
  namespace: seldon
spec:
  selector:
    service: ambassador
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: ambassador-admin
  name: ambassador-admin
  namespace: seldon
spec:
  ports:
  - name: ambassador-admin
    port: 8877
    targetPort: 8877
  selector:
    service: ambassador
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ambassador
  namespace: seldon
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: 'false'
      labels:
        service: ambassador
    spec:
      containers:
      - image: quay.io/datawire/ambassador:0.34.1
        name: ambassador
        env:
        - name: AMBASSADOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        resources:
          limits:
            cpu: 1
            memory: 400Mi
          requests:
            cpu: 200m
            memory: 100Mi
      - image: quay.io/datawire/statsd:0.34.1
        name: statsd
      restartPolicy: Always
      serviceAccountName: ambassador
Read more comments on GitHub >

github_iconTop Results From Across the Web

Ambassador not working on fresh installed kubernetes #437
Describe the bug. The Ambassador container within ambassador pod does not spawn up properly on a local cluster with kubernetes 1.10.2.
Read more >
Running and deployment | Edge Stack
Have Kubernetes run Ambassador Edge Stack as non-root. This may happen by default (e.g., OpenShift) or you can set a securityContext in your...
Read more >
ambassador service stays "pending" - Stack Overflow
Answer is pretty much here: https://serverfault.com/questions/1064313/ambassador-service-stays-pending . After installing a load balancer ...
Read more >
ambassador service stays "pending" - kubernetes - Server Fault
It can't do it with LoadBalancer because you are running a kubernetes ... The installation manifest does not include a configuration file.
Read more >
Integrate Consul with Ambassador Edge Stack on Kubernetes
Configure Ambassador Edge Stack. You must have the AES quickstart completed or properly installed AES for this section to work. The ConsulResolver allows ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found