Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Kibana Security Helm Fails

See original GitHub issue

Chart version: image: “docker.elastic.co/kibana/kibana” imageTag: “8.0.0-SNAPSHOT”

Kubernetes version: 1.21.5

Kubernetes provider: E.g. GKE (Google Kubernetes Engine) GKE

Helm Version: 3.7.1

helm get release output

Eleff:security eleff$ helm -n monitoring get all helm-es-security NAME: helm-es-security LAST DEPLOYED: Sun Feb 20 16:04:21 2022 NAMESPACE: monitoring STATUS: deployed REVISION: 2 USER-SUPPLIED VALUES: clusterName: security createCert: false esConfig: elasticsearch.yml: | xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 nodeGroup: master protocol: https rbac: automountToken: true create: true serviceAccountAnnotations: {} serviceAccountName: “” roles:

master
ingest
data secret: enabled: true password: <REDACTED> secretMounts:
name: elastic-certificates path: /usr/share/elasticsearch/config/certs secretName: elastic-certificates

COMPUTED VALUES: antiAffinity: hard antiAffinityTopologyKey: kubernetes.io/hostname clusterHealthCheckParams: wait_for_status=green&timeout=1s clusterName: security createCert: false enableServiceLinks: true envFrom: [] esConfig: elasticsearch.yml: | xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 esJavaOpts: “” esJvmOptions: {} esMajorVersion: “” extraContainers: [] extraEnvs: [] extraInitContainers: [] extraVolumeMounts: [] extraVolumes: [] fsGroup: “” fullnameOverride: “” healthNameOverride: “” hostAliases: [] httpPort: 9200 image: docker.elastic.co/elasticsearch/elasticsearch imagePullPolicy: IfNotPresent imagePullSecrets: [] imageTag: 8.0.0-SNAPSHOT ingress: annotations: {} className: nginx enabled: false hosts:

host: chart-example.local paths:
- path: / pathtype: ImplementationSpecific tls: [] initResources: {} keystore: [] labels: {} lifecycle: {} masterService: “” maxUnavailable: 1 minimumMasterNodes: 2 nameOverride: “” networkHost: 0.0.0.0 networkPolicy: http: enabled: false transport: enabled: false nodeAffinity: {} nodeGroup: master nodeSelector: {} persistence: annotations: {} enabled: true labels: enabled: false podAnnotations: {} podManagementPolicy: Parallel podSecurityContext: fsGroup: 1000 runAsUser: 1000 podSecurityPolicy: create: false name: “” spec: fsGroup: rule: RunAsAny privileged: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes:
- secret
- configMap
- persistentVolumeClaim
- emptyDir priorityClassName: “” protocol: https rbac: automountToken: true create: true serviceAccountAnnotations: {} serviceAccountName: “” readinessProbe: failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 3 timeoutSeconds: 5 replicas: 3 resources: limits: cpu: 1000m memory: 2Gi requests: cpu: 1000m memory: 2Gi roles:
master
ingest
data schedulerName: “” secret: enabled: true password: <REDACTED> username: <REDACTED> secretMounts:
name: elastic-certificates path: /usr/share/elasticsearch/config/certs secretName: elastic-certificates securityContext: capabilities: drop:
- ALL runAsNonRoot: true runAsUser: 1000 service: annotations: {} enabled: true externalTrafficPolicy: “” httpPortName: http labels: {} labelsHeadless: {} loadBalancerIP: “” loadBalancerSourceRanges: [] nodePort: “” publishNotReadyAddresses: false transportPortName: transport type: ClusterIP sysctlInitContainer: enabled: true sysctlVmMaxMapCount: 262144 terminationGracePeriod: 120 tests: enabled: true tolerations: [] transportPort: 9300 updateStrategy: RollingUpdate volumeClaimTemplate: accessModes:
- ReadWriteOnce resources: requests: storage: 30Gi

HOOKS:

Source: elasticsearch/templates/test/test-elasticsearch-health.yaml

apiVersion: v1 kind: Pod metadata: name: “helm-es-security-hyqrw-test” annotations: “helm.sh/hook”: test “helm.sh/hook-delete-policy”: hook-succeeded spec: securityContext: fsGroup: 1000 runAsUser: 1000 containers:

name: “helm-es-security-yhceh-test” image: “docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT” imagePullPolicy: “IfNotPresent” command:
- “sh”
- “-c”
- | #!/usr/bin/env bash -e curl -XGET --fail ‘security-master:9200/_cluster/health?wait_for_status=green&timeout=1s’ restartPolicy: Never MANIFEST:

Source: elasticsearch/templates/poddisruptionbudget.yaml

apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: “security-master-pdb” spec: maxUnavailable: 1 selector: matchLabels: app: “security-master”

Source: elasticsearch/templates/serviceaccount.yaml

apiVersion: v1 kind: ServiceAccount metadata: name: “security-master” annotations: labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch-8.0.0-SNAPSHOT” app: “security-master”

Source: elasticsearch/templates/secret.yaml

apiVersion: v1 kind: Secret metadata: name: security-master-credentials labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch” app: “security-master” type: Opaque data: username: <REDACTED> password: <REDACTED>

Source: elasticsearch/templates/configmap.yaml

apiVersion: v1 kind: ConfigMap metadata: name: security-master-config labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch” app: “security-master” data: elasticsearch.yml: | xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12

Source: elasticsearch/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: “security-master” labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch-8.0.0-SNAPSHOT” app: “security-master” rules:

apiGroups:
- extensions resources:
- podsecuritypolicies resourceNames:
- “security-master” verbs:
- use

Source: elasticsearch/templates/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: “security-master” labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch-8.0.0-SNAPSHOT” app: “security-master” subjects:

kind: ServiceAccount name: “security-master” namespace: “monitoring” roleRef: kind: Role name: “security-master” apiGroup: rbac.authorization.k8s.io

Source: elasticsearch/templates/service.yaml

kind: Service apiVersion: v1 metadata: name: security-master labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch” app: “security-master” annotations: {} spec: type: ClusterIP selector: release: “helm-es-security” chart: “elasticsearch” app: “security-master” publishNotReadyAddresses: false ports:

name: http protocol: TCP port: 9200
name: transport protocol: TCP port: 9300

Source: elasticsearch/templates/service.yaml

kind: Service apiVersion: v1 metadata: name: security-master-headless labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch” app: “security-master” annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints: “true” spec: clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve

Create endpoints also if the related pod isn’t ready

publishNotReadyAddresses: true selector: app: “security-master” ports:

name: http port: 9200
name: transport port: 9300

Source: elasticsearch/templates/statefulset.yaml

apiVersion: apps/v1 kind: StatefulSet metadata: name: security-master labels: heritage: “Helm” release: “helm-es-security” chart: “elasticsearch” app: “security-master” annotations: esMajorVersion: “8” spec: serviceName: security-master-headless selector: matchLabels: app: “security-master” replicas: 3 podManagementPolicy: Parallel updateStrategy: type: RollingUpdate volumeClaimTemplates:

metadata: name: security-master spec: accessModes:
- ReadWriteOnce resources: requests: storage: 30Gi template: metadata: name: “security-master” labels: release: “helm-es-security” chart: “elasticsearch” app: “security-master” annotations:
  
  configchecksum: 0fb6ccff36e6623c125ec6799220d30697f33f444c341a2352edbc89c9c41a5 spec: securityContext: fsGroup: 1000 runAsUser: 1000 serviceAccountName: “security-master” automountServiceAccountToken: true affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector: matchExpressions:
    - key: app operator: In values:
      - “security-master” topologyKey: kubernetes.io/hostname terminationGracePeriodSeconds: 120 volumes:
  - name: elastic-certificates secret: secretName: elastic-certificates
  - name: esconfig configMap: name: security-master-config enableServiceLinks: true initContainers:
- name: configure-sysctl securityContext: runAsUser: 0 privileged: true image: “docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT” imagePullPolicy: “IfNotPresent” command: [“sysctl”, “-w”, “vm.max_map_count=262144”] resources: {}
containers:
- name: “elasticsearch” securityContext: capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 image: “docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT” imagePullPolicy: “IfNotPresent” readinessProbe: exec: command: - bash - -c - | set -e
```
    # Exit if ELASTIC_PASSWORD in unset
    if [ -z "${ELASTIC_PASSWORD}" ]; then
      echo "ELASTIC_PASSWORD variable is missing, exiting"
      exit 1
    fi

    # If the node is starting up wait for the cluster to be ready (request params: "wait_for_status=green&timeout=1s" )
    # Once it has started only check that the node itself is responding
    START_FILE=/tmp/.es_start_file

    # Disable nss cache to avoid filling dentry cache when calling curl
    # This is required with Elasticsearch Docker using nss < 3.52
    export NSS_SDB_USE_CACHE=no

    http () {
      local path="${1}"
      local args="${2}"
      set -- -XGET -s

      if [ "$args" != "" ]; then
        set -- "$@" $args
      fi

      set -- "$@" -u "elastic:${ELASTIC_PASSWORD}"

      curl --output /dev/null -k "$@" "https://127.0.0.1:9200${path}"
    }

    if [ -f "${START_FILE}" ]; then
      echo 'Elasticsearch is already running, lets check the node is healthy'
      HTTP_CODE=$(http "/" "-w %{http_code}")
      RC=$?
      if [[ ${RC} -ne 0 ]]; then
        echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} https://127.0.0.1:9200/ failed with RC ${RC}"
        exit ${RC}
      fi
      # ready if HTTP code 200, 503 is tolerable if ES version is 6.x
      if [[ ${HTTP_CODE} == "200" ]]; then
        exit 0
      elif [[ ${HTTP_CODE} == "503" && "8" == "6" ]]; then
        exit 0
      else
        echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} https://127.0.0.1:9200/ failed with HTTP code ${HTTP_CODE}"
        exit 1
      fi

    else
      echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
      if http "/_cluster/health?wait_for_status=green&timeout=1s" "--fail" ; then
        touch ${START_FILE}
        exit 0
      else
        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
        exit 1
      fi
    fi
```
  failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 3 timeoutSeconds: 5 ports:
  - name: http containerPort: 9200
  - name: transport containerPort: 9300 resources: limits: cpu: 1000m memory: 2Gi requests: cpu: 1000m memory: 2Gi env:
    - name: node.name valueFrom: fieldRef: fieldPath: metadata.name
    - name: cluster.initial_master_nodes value: “security-master-0,security-master-1,security-master-2,”
    - name: node.roles value: “master,ingest,data,”
    - name: discovery.seed_hosts value: “security-master-headless”
    - name: cluster.name value: “security”
    - name: network.host value: “0.0.0.0”
    - name: ELASTIC_PASSWORD valueFrom: secretKeyRef: name: security-master-credentials key: password volumeMounts:
    - name: “security-master” mountPath: /usr/share/elasticsearch/data
    - name: elastic-certificates mountPath: /usr/share/elasticsearch/config/certs
    - name: esconfig mountPath: /usr/share/elasticsearch/config/elasticsearch.yml subPath: elasticsearch.yml

NOTES:

Watch all cluster members come up. $ kubectl get pods --namespace=monitoring -l app=security-master -w
Retrieve elastic user’s password. $ kubectl get secrets --namespace=monitoring security-master-credentials -ojsonpath=‘{.data.password}’ | base64 -d
Test cluster health using Helm test. $ helm --namespace=monitoring test helm-es-security

Describe the bug:

Kibana pod in crashloop. Seems to be due to this Elastic patch:

https://github.com/elastic/kibana/pull/122722

We first deployed the Kibana security example in dev a month ago, everything worked great. We tried again yesterday on a new cluster, and it’s failing.

Steps to reproduce:

Deploy the following helm chart:

https://github.com/elastic/helm-charts/tree/main/kibana/examples/security

Expected behavior: Kibana should run

Provide logs and/or server output (if relevant):

kubectl -n monitoring logs helm-kibana-security-kibana-5cfb854b85-sh95r [2022-02-21T07:12:28.619+00:00][INFO ][plugins-service] Plugin “metricsEntities” is disabled. [2022-02-21T07:12:28.723+00:00][FATAL][root] Error: [config validation of [elasticsearch].username]: value of “elastic” is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html at ensureValidConfiguration (/usr/share/kibana/src/core/server/config/ensure_valid_configuration.js:25:11) at Server.preboot (/usr/share/kibana/src/core/server/server.js:160:5) at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:14) at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9) at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)

FATAL Error: [config validation of [elasticsearch].username]: value of “elastic” is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html

Be careful to obfuscate every secrets (credentials, token, public IP, …) that could be visible in the output before copy-pasting

Any additional context:

Issue Analytics

State:
Created 2 years ago
Comments:7 (2 by maintainers)

Top GitHub Comments

1reaction

jmlrtcommented, Mar 8, 2022

@yitzchake, 8.x products are not yet supported by the charts.

1reaction

howardshawcommented, Feb 22, 2022

+1 same issue

error msg as below

[2022-02-22T03:42:33.474+00:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
[2022-02-22T03:42:33.506+00:00][FATAL][root] Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html
    at ensureValidConfiguration (/usr/share/kibana/src/core/server/config/ensure_valid_configuration.js:25:11)
    at Server.preboot (/usr/share/kibana/src/core/server/server.js:160:5)
    at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)

 FATAL  Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html