Cannot mount elasticsearch keystore in pod, device busy

I’m trying to mount the elasticsearch keystore per the documentation but I keep getting the error below. I’ve verified the the elasticsearch keystore file is valid (by adding it to a test continer and testing the values).

Error:

Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy

Steps taken:

1. Command to create the secret: kubectl create secret generic elasticsearch-keystore --from-file=./elasticsearch.keystore

2. SecretMounts in my yaml configuration:

  secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs
  - name: elastic-license
    secretName: elastic-license
    path: /usr/share/elasticsearch/config/license
  - name: elasticsearch-keystore
    secretName: elasticsearch-keystore
    path: /usr/share/elasticsearch/config/elasticsearch.keystore
    subPath: elasticsearch.keystore 

Is there something I am missing?

[Crazybus]

@JanKowalik Thank you for providing the extra details. I think that the issue you are running into is because you have the $ELASTIC_PASSWORD set but aren’t adding it to your keystore as the bootstrap password. The docker image startup script tries to add this to the keystore when it is set.

You want to add it with:

elasticsearch-keystore add -x bootstrap.password

Once https://github.com/elastic/helm-charts/pull/154 is finished off there won’t be any need to manually create and update the keystore anymore.

[Crazybus]

Well that was a fun journey. I found it though!

In https://github.com/elastic/elasticsearch/pull/41701 the internal format for the keystore was bumped from version 3 to version 4. On startup Elasticsearch notices this and attempts to upgrade the format of the keystore. All mounted secrets and configmaps in Kubernetes are always readonly. So when it tries to make the change you get the error as seen above.

So the fix is to either:

1. Make sure you are creating the keystore with the right version of Elasticsearch. So if you are using 7.1.1 in the cluster you should use the same version when creating the keystore
2. If you already have an existing keystore you should run elasticsearch-keystore upgrade with the right Elasticsearch version to get it upgraded to the right format.