[kibana] initContainer: configure-kibana-token: Back-off restarting failed container:

Chart version: 8.4.1 (from the main branch in this point in history)

Kubernetes version: 1.21

Kubernetes provider: GKE (Google Kubernetes Engine)

Describe the bug: Kibana’s initContainer configure-kibana-token keep crashing forever.

Steps to reproduce:

1. Deploy elasticsearch 8 helm chart and enable security features (username/password + SSL)
2. Deploy kibana 8 helm chart and reference the appropriate elasticsearch credentials and certificates secrets
3. After kibana’s pod successfully runs for the first time, delete it
4. the new kibana pod will have its initContainer configure-kibana-token crashing forever.

Expected behavior: the new kibana pod will have its initContainer configure-kibana-token completes successfully.

Provide logs and/or server output (if relevant): configure-kibana-token initContainer logs before crashing:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 409 Conflict

this init container creates a token for kibana’s service account and saves it for kibana’s actual container.

if i run a similar command from within the elasticsearch pods: curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD -XPOST https://localhost:9200/_security/service/elastic/kibana/credential/token/mykibana8-kibana?pretty I get the following response:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "version_conflict_engine_exception",
        "reason" : "[service_account_token-elastic/kibana/mykibana8-kibana]: version conflict, document already exists (current version [1])",
        "index_uuid" : "zwuGuJcSS2OUs1ClsFdThB",
        "shard" : "0",
        "index" : ".security-7"
      }
    ],
    "type" : "version_conflict_engine_exception",
    "reason" : "[service_account_token-elastic/kibana/mykibana8-kibana]: version conflict, document already exists (current version [1])",
    "index_uuid" : "zwuGuJcSS2OUs1ClsFdThB",
    "shard" : "0",
    "index" : ".security-7"
  },
  "status" : 409
}

if i manually delete that token: curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD -XDELETE https://localhost:9200/_security/service/elastic/kibana/credential/token/mykibana8-kibana?pretty

{
  "found" : true
}

and then the pod can start. but again, if that pod dies, the next one will get stuck the same way.

[jmlrt]

Indeed, I think the best solution is to use a pre-install job that:

1. delete the existing token matching the Helm chart release name
2. create a new token matching the Helm chart release name
3. encode the token in base64
4. call the K8S api to create a secret with the base64 encoded token

Then mount the secret into all kibana pods and finaly remove the token + secret in a post-delete job.

I was already able to test the secret creation from a pod using k8s api.

Now, I’m trying to write a node JS script to do all the pre-install steps. I’m struggling a bit since that’s my first time looking at node and javascript 😉. This would have been a lot easier in Python for example, however, node is the only language interpreter installed into the Kibana Docker image and there is no jq to parse json so using bash is not an option 😦

PR in progress => https://github.com/elastic/helm-charts/pull/1720 (still a few things to fix 🤞🏻)

[jmlrt]

Thanks, @mdnfiras for submitting this issue and PR 👍🏻

Indeed, it seems I forgot to handle the case where a pod is destroyed in https://github.com/elastic/helm-charts/pull/1679.