Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Docker Desktop k8s: AccessDeniedException /usr/share/elasticsearch/data/nodes

See original GitHub issue

Chart version: ?

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:02:12Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes provider: E.g. GKE (Google Kubernetes Engine)

Helm Version:

Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

helm get release output ?

Describe the bug:

Can’t run ES using helm on Docker Desktop (x64 Windows 10 Pro).

Tried solutions mentioned in #258 and #137 to no avail.

Steps to reproduce:

install Docker Desktop
Enable kubernetes mode
install helm 2.16
enable tiller
run helm repo add elastic https://helm.elastic.co
run helm install --name elasticsearch elastic/elasticsearch

Expected behavior:

running ES

Provide logs and/or server output (if relevant):

>kubectl get all


NAME                         READY   STATUS    RESTARTS   AGE
pod/elasticsearch-master-0   0/1     Error     1          21s
pod/elasticsearch-master-1   0/1     Pending   0          21s
pod/elasticsearch-master-2   0/1     Pending   0          21s

NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch-master            ClusterIP   10.109.99.150   <none>        9200/TCP,9300/TCP   21s
service/elasticsearch-master-headless   ClusterIP   None            <none>        9200/TCP,9300/TCP   21s
service/kubernetes                      ClusterIP   10.96.0.1       <none>        443/TCP             3d1h

NAME                                    READY   AGE
statefulset.apps/elasticsearch-master   0/3     21s


>kubectl logs pod/elasticsearch-master-0


OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-11-29T12:48:56,326Z", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.1.jar:7.4.1]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.1.jar:7.4.1]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:614) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:209) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]",

>kubectl describe pod/elasticsearch-master-0

Name:           elasticsearch-master-0
Namespace:      default
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Fri, 29 Nov 2019 14:48:43 +0200
Labels:         app=elasticsearch-master
                chart=elasticsearch
                controller-revision-hash=elasticsearch-master-56cf946ddc
                heritage=Tiller
                release=elasticsearch
                statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations:    <none>
Status:         Running
IP:             10.1.1.66
IPs:            <none>
Controlled By:  StatefulSet/elasticsearch-master
Init Containers:
  configure-sysctl:
    Container ID:  docker://cc7dc88de5a0a3bb782d81e1c0a817c34c41b261255369caaee6311fce9b484e
    Image:         docker.elastic.co/elasticsearch/elasticsearch:7.4.1
    Image ID:      docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:88c2ee30115f378b8f7e66662ec26bca0c8778c69096bee6b161128ce833585f
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 29 Nov 2019 14:48:46 +0200
      Finished:     Fri, 29 Nov 2019 14:48:46 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9d5l2 (ro)
Containers:
  elasticsearch:
    Container ID:   docker://1f637eee1efe3dc170556fb03184ed04776e2ab8b3506e03b1a62b6e3598412c
    Image:          docker.elastic.co/elasticsearch/elasticsearch:7.4.1
    Image ID:       docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:88c2ee30115f378b8f7e66662ec26bca0c8778c69096bee6b161128ce833585f
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 29 Nov 2019 14:50:34 +0200
      Finished:     Fri, 29 Nov 2019 14:50:38 +0200
    Ready:          False
    Restart Count:  4
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:      100m
      memory:   2Gi
    Readiness:  exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file

http () {
    local path="${1}"
    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
    else
      BASIC_AUTH=''
    fi
    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}

if [ -f "${START_FILE}" ]; then
    echo 'Elasticsearch is already running, lets check the node is healthy'
    http "/"
else
    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
        touch ${START_FILE}
        exit 0
    else
        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
        exit 1
    fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      node.name:                     elasticsearch-master-0 (v1:metadata.name)
      cluster.initial_master_nodes:  elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
      discovery.seed_hosts:          elasticsearch-master-headless
      cluster.name:                  elasticsearch
      network.host:                  0.0.0.0
      ES_JAVA_OPTS:                  -Xmx1g -Xms1g
      node.data:                     true
      node.ingest:                   true
      node.master:                   true
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9d5l2 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  elasticsearch-master:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-master-elasticsearch-master-0
    ReadOnly:   false
  default-token-9d5l2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-9d5l2
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                    From                     Message
  ----     ------            ----                   ----                     -------
  Warning  FailedScheduling  2m56s (x2 over 2m56s)  default-scheduler        pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         2m54s                  default-scheduler        Successfully assigned default/elasticsearch-master-0 to docker-desktop
  Normal   Pulled            2m52s                  kubelet, docker-desktop  Container image "docker.elastic.co/elasticsearch/elasticsearch:7.4.1" already present on machine
  Normal   Created           2m51s                  kubelet, docker-desktop  Created container configure-sysctl
  Normal   Started           2m51s                  kubelet, docker-desktop  Started container configure-sysctl
  Normal   Pulled            64s (x5 over 2m51s)    kubelet, docker-desktop  Container image "docker.elastic.co/elasticsearch/elasticsearch:7.4.1" already present on machine
  Normal   Created           64s (x5 over 2m51s)    kubelet, docker-desktop  Created container elasticsearch
  Normal   Started           63s (x5 over 2m50s)    kubelet, docker-desktop  Started container elasticsearch
  Warning  BackOff           58s (x7 over 2m41s)    kubelet, docker-desktop  Back-off restarting failed container


>kubectl get pv


NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                 STORAGECLASS   REASON   AGE
pvc-9254b47b-12a6-11ea-8ba5-00155d005f16   30Gi       RWO            Delete           Bound    default/elasticsearch-master-elasticsearch-master-0   hostpath                9m52s
pvc-9259df87-12a6-11ea-8ba5-00155d005f16   30Gi       RWO            Delete           Bound    default/elasticsearch-master-elasticsearch-master-1   hostpath                9m38s
pvc-925f6b4e-12a6-11ea-8ba5-00155d005f16   30Gi       RWO            Delete           Bound    default/elasticsearch-master-elasticsearch-master-2   hostpath                9m38s


>kubectl get pvc


NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-master-elasticsearch-master-0   Bound    pvc-9254b47b-12a6-11ea-8ba5-00155d005f16   30Gi       RWO            hostpath       9m58s
elasticsearch-master-elasticsearch-master-1   Bound    pvc-9259df87-12a6-11ea-8ba5-00155d005f16   30Gi       RWO            hostpath       9m58s
elasticsearch-master-elasticsearch-master-2   Bound    pvc-925f6b4e-12a6-11ea-8ba5-00155d005f16   30Gi       RWO            hostpath       9m58s

Any additional context:

Issue Analytics

State:
Created 4 years ago
Reactions:1
Comments:13 (9 by maintainers)

Top GitHub Comments

3reactions

salaboycommented, Dec 12, 2019

I can confirm that this is still failing in Kubernetes KIND as well

1reaction

jmlrtcommented, Jan 8, 2020

@chattarajoy thanks to your link to kubernetes-sigs/kind#830, I was able to test a workaround using Rancher Local Path Provisioner while waiting for the fix on kubernetes-sigs/kind side in their next release.

More details in https://github.com/elastic/helm-charts/issues/429#issuecomment-571969666.

Top Results From Across the Web

docker - Elastic search AccessDeniedException[/usr/share ...

This error is occurring due to permission issue on specified path. Change the path from /usr/share/elasticsearch/data to ...

Deploy on Kubernetes - Docker Documentation

Deploy on Kubernetes. Docker Desktop includes a standalone Kubernetes server and client, as well as Docker CLI integration that runs on your machine....

How Kubernetes works under the hood with Docker Desktop

Docker Desktop makes developing applications for Kubernetes easy. It provides a smooth Kubernetes setup experience by hiding the complexity ...

Dockershim not needed: Docker Desktop with Kubernetes 1.24+

As a developer using Docker Desktop, do I need to care which container runtime implementation is used in production? No, the container “runtime” ......

Creating Kubernetes Extensions in Docker Desktop

Learn what it takes to create Kubernetes Extensions in Docker Desktop! In this guest blog, James Spurin recounts how he created the k9s ......