Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

502 error when accessing ingress

See original GitHub issue

Chart version: Release 7.1.1

Kubernetes version:

Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.1”, GitCommit:“eec55b9ba98609a46fee712359c7b5b365bdd920”, GitTreeState:“clean”, BuildDate:“2018-12-13T10:39:04Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“windows/amd64”} Server Version: version.Info{Major:“1”, Minor:“10”, GitVersion:“v1.10.6”, GitCommit:“a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e”, GitTreeState:“clean”, BuildDate:“2018-07-26T10:04:08Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“linux/amd64”}

Kubernetes provider: E.g. GKE (Google Kubernetes Engine)

KOPS on AWS

Helm Version:

Client: &version.Version{SemVer:“v2.11.0”, GitCommit:“2e55dbe1fdb5fdb96b75ff144a339489417b146b”, GitTreeState:“clean”} Server: &version.Version{SemVer:“v2.11.0”, GitCommit:“2e55dbe1fdb5fdb96b75ff144a339489417b146b”, GitTreeState:“clean”}

helm get release output

REVISION: 1
RELEASED: Fri Jun  7 14:44:46 2019
CHART: kibana-7.1.0
USER-SUPPLIED VALUES:
elasticsearchHosts: https://elasticsearch-master:9200
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
ingress:
  annotations: {}
  enabled: true
  hosts:
  - xxx.io
kibanaConfig:
  kibana.yml: |
    server.host: kibana.xxx.io
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
protocol: http
secretMounts:
- name: elastic-certificate-pem
  path: /usr/share/kibana/config/certs
  secretName: elastic-certificate-pem

COMPUTED VALUES:
affinity: {}
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
elasticsearchHosts: https://elasticsearch-master:9200
elasticsearchURL: ""
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
fullnameOverride: ""
healthCheckPath: /app/kibana
httpPort: 5601
image: docker.elastic.co/kibana/kibana
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.1.0
ingress:
  annotations: {}
  enabled: true
  hosts:
  - kibana.xxx.io
  path: /
  tls: []
kibanaConfig:
  kibana.yml: |
    server.host: kibana.xxx.io
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
maxUnavailable: 1
nameOverride: ""
nodeSelector: {}
podSecurityContext: {}
priorityClassName: ""
protocol: http
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 1000m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 500m
secretMounts:
- name: elastic-certificate-pem
  path: /usr/share/kibana/config/certs
  secretName: elastic-certificate-pem
service:
  port: 5601
  type: ClusterIP
serviceAccount: ""
tolerations: []
updateStrategy:
  type: Recreate

HOOKS:
MANIFEST:

---
# Source: kibana/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-kibana-config
  labels:
    app: kibana
    release: "kibana"
data:
  kibana.yml: |
    server.host: kibana.xxx.io
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
---
# Source: kibana/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: "kibana"
    heritage: Tiller
spec:
  type: ClusterIP
  ports:
    - port: 5601
      protocol: TCP
      name: http
      targetPort: 5601
  selector:
    app: kibana
    release: "kibana"
---
# Source: kibana/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: "kibana"
spec:
  replicas: 1
  strategy:
    type: Recreate

  selector:
    matchLabels:
      app: kibana
      release: "kibana"
  template:
    metadata:
      labels:
        app: kibana
        release: "kibana"
      annotations:

        configchecksum: d6304589bb11d4439446b7290fbda4717d8e393c9e55733751380c6e3bf713e
    spec:
      volumes:
        - name: elastic-certificate-pem
          secret:
            secretName: elastic-certificate-pem
        - name: kibanaconfig
          configMap:
            name: kibana-kibana-config
      containers:
      - name: kibana
        image: "docker.elastic.co/kibana/kibana:7.1.0"
        env:
          - name: ELASTICSEARCH_HOSTS
            value: "https://elasticsearch-master:9200"
          - name: ELASTICSEARCH_USERNAME
            valueFrom:
              secretKeyRef:
                key: username
                name: elastic-credentials
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: elastic-credentials

        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                http () {
                    local path="${1}"
                    set -- -XGET -s --fail

                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    fi

                    curl -k "$@" "http://localhost:5601${path}"
                }

                http "/app/kibana"
        ports:
        - containerPort: 5601
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 500m

        volumeMounts:
          - name: elastic-certificate-pem
            mountPath: /usr/share/kibana/config/certs
          - name: kibanaconfig
            mountPath: /usr/share/kibana/config/kibana.yml
            subPath: kibana.yml
---
# Source: kibana/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: kibana
    heritage: Tiller
spec:
  rules:
    - host: kibana.xxx.io
      http:
        paths:
          - path: /
            backend:
              serviceName: kibana-kibana
              servicePort: 5601

Describe the bug:

When I try access the ingress that is created above I get a 502 error from nginx.

Steps to reproduce:

Install the secure version of the ES helm chart.
Install kibana helm chart with a pem ca cert that was used to create the ES chart.
Go to the ingress.

Expected behavior:

Able to access kibana from the above ingress

Provide logs and/or server output (if relevant):

Any additional context:

Kibana appears to be working and connects to the Elastic. All my pods are green. When I do a port forward to the service or the deployment it loads fine.

kubectl -n logging port-forward svc/kibana-kibana 5601

It’s the ingress that isn’t working. If I try a curl from inside a pod, I also get a connection refused.

curl kibana-kibana.logging.svc.cluster.local:5601 curl: (7) Failed to connect to kibana-kibana.logging.svc.cluster.local port 5601: Connection refused