Readiness probe failed: Error: Got HTTP code 503 but expected a 200

Chart version: 7.7.1 Kubernetes version: 1.16 Kubernetes provider: E.g. GKE (Google Kubernetes Engine) EKS Helm Version: 2.16.10

helm get release output

e.g. helm get elasticsearch (replace elasticsearch with the name of your helm release)

Be careful to obfuscate every secrets (credentials, token, public IP, …) that could be visible in the output before copy-pasting.

If you find some secrets in plain text in helm get release output you should use Kubernetes Secrets to managed them is a secure way (see Security Example).

Output of helm get release

helm get kibana

REVISION: 1
RELEASED: Thu Aug 13 20:14:45 2020
CHART: kibana-7.7.1
USER-SUPPLIED VALUES:
{}

COMPUTED VALUES:
affinity: {}
elasticsearchHosts: http://elasticsearch-master:9200
elasticsearchURL: ""
envFrom: []
extraContainers: ""
extraEnvs:
- name: NODE_OPTIONS
  value: --max-old-space-size=1800
extraInitContainers: ""
fullnameOverride: ""
healthCheckPath: /app/kibana
httpPort: 5601
image: docker.elastic.co/kibana/kibana
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.7.1
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
kibanaConfig: {}
labels: {}
lifecycle: {}
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext:
  fsGroup: 1000
priorityClassName: ""
protocol: http
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 800m
    memory: 1Gi
  requests:
    cpu: 800m
    memory: 1Gi
secretMounts: []
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
serverHost: 0.0.0.0
service:
  annotations: {}
  labels: {}
  loadBalancerSourceRanges: []
  nodePort: ""
  port: 5601
  type: ClusterIP
serviceAccount: ""
tolerations: []
updateStrategy:
  type: Recreate

HOOKS:
MANIFEST:

---
# Source: kibana/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kibana-kibana
  labels: 
    app: kibana
    release: "kibana"
    heritage: Tiller
spec:
  type: ClusterIP
  ports:
    - port: 5601
      protocol: TCP
      name: http
      targetPort: 5601
  selector:
    app: kibana
    release: "kibana"
---
# Source: kibana/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-kibana
  labels: 
    app: kibana
    release: "kibana"
    heritage: Tiller
spec:
  replicas: 1
  strategy:
    type: Recreate
    
  selector:
    matchLabels:
      app: kibana
      release: "kibana"
  template:
    metadata:
      labels:
        app: kibana
        release: "kibana"
      annotations:
        
    spec:
      securityContext:
        fsGroup: 1000
        
      volumes:
      containers:
      - name: kibana
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
          
        image: "docker.elastic.co/kibana/kibana:7.7.1"
        imagePullPolicy: "IfNotPresent"
        env:
          - name: ELASTICSEARCH_HOSTS
            value: "http://elasticsearch-master:9200"
          - name: SERVER_HOST
            value: "0.0.0.0"
          - name: NODE_OPTIONS
            value: --max-old-space-size=1800
          
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
          
          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                http () {
                    local path="${1}"
                    set -- -XGET -s --fail -L

                    if [ -n "${ELASTICSEARCH_USERNAME}" ] && [ -n "${ELASTICSEARCH_PASSWORD}" ]; then
                      set -- "$@" -u "${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD}"
                    fi

                    STATUS=$(curl --output /dev/null --write-out "%{http_code}" -k "$@" "http://localhost:5601${path}")
                    if [[ "${STATUS}" -eq 200 ]]; then
                      exit 0
                    fi

                    echo "Error: Got HTTP code ${STATUS} but expected a 200"
                    exit 1
                }

                http "/app/kibana"
        ports:
        - containerPort: 5601
        resources:
          limits:
            cpu: 800m
            memory: 1Gi
          requests:
            cpu: 800m
            memory: 1Gi
          
        volumeMounts:

Describe the bug:

kubectl describe pod kibana-kibana-7458222222-2222

  Warning  Unhealthy  26m (x5 over 27m)  kubelet, ip-101-10-161-126.us-west-2.compute.internal  Readiness probe failed: Error: Got HTTP code 000 but expected a 200
  Warning  Unhealthy  26m                kubelet, ip-101-10-161-126.us-west-2.compute.internal  Readiness probe failed: Error: Got HTTP code 503 but expected a 200

Steps to reproduce:

helm install --name elasticsearch ./elasticsearch --namespace elk 2. helm install --name kibana ./kibana --namespace elk 3.

Expected behavior:

Provide logs and/or server output (if relevant):

Be careful to obfuscate every secrets (credentials, token, public IP, …) that could be visible in the output before copy-pasting

Any additional context:

[Ankitchandre]

you can change the value of healthCheckPath: from “/app/kibana” to “/api/status” in values.yaml . This should pass the readiness probe error .

[abdennour]

But if i have to change healthCheckPath to a new value , why is not the default value ? Guys, please check best practices implemented in som helm git repos ( e.g: bitnami,… etc) and mimic them. Overall, good job! thanks!