AccessDeniedException /usr/share/elasticsearch/data/nodes

Chart version: version: 7.0.1-alpha1 Kubernetes version: 1.11 Kubernetes provider: E.g. GKE (Google Kubernetes Engine) IBM cloud Helm Version: 2.9.1 Values.yaml:

---
clusterName: "elasticsearch"
nodeGroup: "master"

# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""

# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
  master: "true"
  ingest: "true"
  data: "true"

replicas: 3
minimumMasterNodes: 2

esMajorVersion: 7

# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig: {}
#  elasticsearch.yml: |
#    key:
#      nestedkey: value
#  log4j2.properties: |
#    key = value

# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: []
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
#  - name: elastic-certificates
#    secretName: elastic-certificates
#    path: /usr/share/elasticsearch/config/certs

image: docker.elastic.co/elasticsearch/elasticsearch
imageTag: 7.0.1
imagePullPolicy: IfNotPresent

podAnnotations: {}
  # iam.amazonaws.com/role: es-cluster

esJavaOpts: "-Xmx1g -Xms1g"

resources:
  requests:
    cpu: "100m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

initResources: {}
  # limits:
  #   cpu: "25m"
  #   # memory: "128Mi"
  # requests:
  #   cpu: "25m"
  #   memory: "128Mi"

networkHost: "0.0.0.0"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 30Gi

persistence:
  enabled: true
  annotations: {}

extraVolumes: []
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: []
  # - name: extras
  #   mountPath: /usr/share/extras
  #   readOnly: true

extraInitContainers: []
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"

# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"

protocol: http
httpPort: 9200
transportPort: 9300

updateStrategy: RollingUpdate

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

 # GroupID for the elasticsearch user. The official elastic docker images always have the id of 1000
fsGroup: 1000

# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"

nodeSelector: {}
tolerations: []

# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

nameOverride: ""
fullnameOverride: ""

Describe the bug: I initially had some issues with the image and tag due to “” so i did this: to fix that: image: docker.elastic.co/elasticsearch/elasticsearch:{{ .Values.imageTag }}

but now the kubernetes pods are in a crashloopbackoff state with this in their logs:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. {“type”: “server”, “timestamp”: “2019-05-16T21:50:15,565+0000”, “level”: “WARN”, “component”: “o.e.b.ElasticsearchUncaughtExceptionHandler”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “uncaught exception in thread [main]” , “stacktrace”: [“org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];”, “at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.1.jar:7.0.1]”, “at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.1.jar:7.0.1]”, “Caused by: org.elasticsearch.ElasticsearchException: failed to bind service”, “at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]”, “… 6 more”, “Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes”, “at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]”, “at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]”, “at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]”, “at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]”, “at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]”, “at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]”, “at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]”, “at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:207) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.node.Node.<init>(Node.java:271) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]”, “at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]”, “… 6 more”] } Steps to reproduce:

Expected behavior:

Provide logs and/or server output (if relevant):

Any additional context:

[ceastman-ibm]

@Crazybus i think this has helped - at least for my IBM cloud 1.13 kube cluster: https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#file_app_failures

in the initContainer section of the statefulset.yaml i added: chown -R 1000:1000 /usr/share/elasticsearch/data

this seems to have gotten around the access denied issue.

[dzmitry-lahoda]

i have checked official es operator and used

      securityContext:
        fsGroup: 1000

from it. it worked well.