Kibana Chart with "kibana_system" user: readinessProbe never succeeds

Chart version: 7.12.0 Kubernetes version: 1.18.2 Kubernetes provider: E.g. GKE (Google Kubernetes Engine) Kubernetes is set up with Rancher

Helm Version: v3

helm get release output The environment is air-gapped but I think the bug is simple enough to reproduce.

Describe the bug: When I set up kibana to use the kibana_system user (as recommended for production) the readinessProbe never succeeds.

This is because kibana settings and the readinessProbe use the same user and password ( ELASTICSEARCH_USERNAMEand ELASTICSEARCH_PASSWORD) to connect to elasticsearch and kibana, but the kibana_system user lacks permission to access kibana. Therefore the call to app/kibana returns a 403 HTTP Status Code, but the readinessProbe expects a 200, as the code snippet shows:

https://github.com/elastic/helm-charts/blob/b3a4b753a564811713f1a5ff04669baa3a4adfd7/kibana/templates/deployment.yaml#L124-L127

Once I manually edit the readinessProbe to accept a 403 code everything works fine

Steps to reproduce:

1. Set kibana_system user as ELASTICSEARCH_USERNAME in kibana helm template
2. Deploy kibana helm template
3. Watch kibana readinessProbe failing with Error: Got HTTP code 403 but expected a 200

Expected behavior:

The readinessProbe should recognise that kibana is fully up and running and succeed if the kibana_system user is used.

One could either accept 403, too or just access / (which then gets redirected to /login?next=%2F ) and evaluate the http status code there (which is 200 for an unauthenticated user as well)

[stijnbrouwers]

Same issue here and resolved it by setting

healthCheckPath: /api/status

as proposed by @josecu08

[josecu08]

My workaround was to use another endpoint for healthcheck. I’m using /api/status instead of /app/kibana.