question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Kibana health check fails when Auth is enabled because readiness probe 302's to login

See original GitHub issue

Chart version:

7.5.1

Kubernetes version:

1.13.1-eks

Kubernetes provider: E.g. GKE (Google Kubernetes Engine)

EKS

Helm Version:

2.14.3

helm get release output

e.g. helm get elasticsearch (replace elasticsearch with the name of your helm release)

Be careful to obfuscate every secrets (credentials, token, public IP, …) that could be visible in the output before copy-pasting.

If you find some secrets in plain text in helm get release output you should use Kubernetes Secrets to managed them is a secure way (see Security Example).

Output of helm get release

Describe the bug:

Kibana never becomes healthy when Auth is enabled (by setting ELASTICSEARCH_USERNAME and ELASTICSEARCH_PASSWORD environment variables). The readiness probe includes:

              STATUS=$(curl --output /dev/null --write-out "%{http_code}" -k "$@" "http://localhost:5601${path}")
              if [[ "${STATUS}" -eq 200 ]]; then
                exit 0
              fi

However this does not 200, but instead 302’s to /login:

bash-4.2$ curl http://localhost:5601/app/kibana -vvv
* About to connect() to localhost port 5601 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5601 (#0)
> GET /app/kibana HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5601
> Accept: */*
> 
< HTTP/1.1 302 Found
< location: /login?next=%2Fapp%2Fkibana
< cache-control: no-cache
< content-length: 0
< Date: Fri, 10 Jan 2020 20:34:29 GMT
< Connection: keep-alive

This is confirmed with logs:

{"type":"response","@timestamp":"2020-01-10T20:39:43Z","tags":[],"pid":8,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":302,"responseTime":24,"contentLength":9},"message":"GET /app/kibana 302 24ms - 9.0B"}

Any additional context:

It might be sufficient to just have a health check that instead of checks for an explicit 200, makes sure its less than a 400 or 500.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:10 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
SergKcommented, Jan 13, 2020

Hi @justinturpin , this PR https://github.com/elastic/helm-charts/pull/405 should resolve your issue

0reactions
jgatocommented, Apr 20, 2020

the problem came from other error. I have not properly configured the CA from Elasticsearch into Kibana. My fault, because I created my own self-signed certificates not been an expert on that. So my CA pem was empty.

But still have this doubt, is it secure to have passwords in the env variables?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Readiness check failure when enabling OIDC authentication
I'm trying to enable OIDC authentication in elastic/kibana deployed by the operator, having enabled the platinum subscription trial.
Read more >
How to Troubleshoot and Address Liveness / Readiness ...
Liveness / Readiness probe failure are caused by Jenkins being not responsive to a health check - currently done https://$POD_IP:8080/$MASTER_NAME/login.
Read more >
Elasticsearch pod readiness probe fails with "message"
If this failed inside the ES container: curl -k https://127.0.0.0:9200 it's clear that the ES is not "live". This could mean from K8s ......
Read more >
Kibana - Liveness probe / health check
There is a Kibana deployed on k8s with a liveness probe, using httpGet/https with basic auth httpHeaders. It allows to check kibana is...
Read more >
Access to Fastly's web interface controls About the All services ...
Single Sign On lets you manage user authentication by enabling single ... Fastly performs health checks on your origin server based on the ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found