1.16/candidate - Cluster volume permssion problems on non-master nodes

I have a 3 node cluster. I have set the --allow-privileged flag in /args/kube-apiserver (unsure if that’s related). Addons enabled: dns, storage.

I setup the elasticsearch operator (1.0 beta) and setup a cluster just fine - eventually turns green (healthy). I try setting up a strimzi operator (0.14.0) to manage kafka cluster and it only gets as far as creating the zookeeper pods and only the pod on the master node succeeds. Pod logs complains about permission denied accessing /var/lib/zookeeper/

After much fussing around I tracked down the issue to be permissions of the files on disk on the nodes themselves in /var/snap/microk8s/common/default-storage On master:

drwxr-xr-x  5 root     root     4096 Oct 18 17:24 ..
drwxrwxrwx  2 root     root     4096 Oct 18 17:53 dev-data-ws-zookeeper-0-pvc-f9ad9e3b-4a82-4e33-99be-e4c1d0ac7c72
drwxrwxrwx  2 root     root     4096 Oct 18 17:53 dev-data-ws-zookeeper-1-pvc-6fcb6136-73f5-4102-a215-53f452b8f2fe
drwxrwxrwx  4 root     root     4096 Oct 18 17:53 dev-data-ws-zookeeper-2-pvc-adeeebcd-2470-413e-ab9d-0b99bb18025d
drwxrwxrwx  2 root     root     4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-coordinating-nodes-0-pvc-4edf829e-f42a-4ab1-958c-7efe7beb7774
drwxrwxrwx  2 root     root     4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-coordinating-nodes-1-pvc-79529357-0f6d-495b-a1b2-e83be0085016
drwxrwxrwx  3 rsiadmin rsiadmin 4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-data-nodes-0-pvc-0303c3c6-4e9c-4704-b4b3-91383de91c62
drwxrwxrwx  2 root     root     4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-data-nodes-1-pvc-1bfde038-4f42-4502-b4f8-cd57de3c0e86
drwxrwxrwx  2 root     root     4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-data-nodes-2-pvc-4b12de2e-be02-44d2-861f-06325bbaa28c
drwxrwxrwx  2 root     root     4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-master-nodes-0-pvc-dce03d45-736f-4b2d-8c2d-72bc85794dbf
drwxrwxrwx  2 root     root     4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-master-nodes-1-pvc-f84aae88-ef12-4a1a-9628-8ddd6212723b
drwxrwxrwx  3 rsiadmin rsiadmin 4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-master-nodes-2-pvc-420137ef-9049-43d3-9dba-72a322c5a41b

On the non-master nodes:

drwxr-xr-x  2 root     root     4096 Oct 18 17:53 dev-data-ws-zookeeper-0-pvc-f9ad9e3b-4a82-4e33-99be-e4c1d0ac7c72
drwxr-xr-x  2 root     root     4096 Oct 18 17:14 dev-data-ws-zookeeper-1-pvc-2c53768b-0341-443f-838f-dadf70308308
drwxr-xr-x  3 rsiadmin rsiadmin 4096 Oct 18 17:27 dev-elasticsearch-data-ws-es-coordinating-nodes-0-pvc-d77e0102-74c3-4dcd-a0dd-9d9fddbf93ca
drwxr-xr-x  3 rsiadmin rsiadmin 4096 Oct 18 17:33 dev-elasticsearch-data-ws-es-coordinating-nodes-1-pvc-79529357-0f6d-495b-a1b2-e83be0085016
drwxr-xr-x  3 rsiadmin rsiadmin 4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-data-nodes-1-pvc-1bfde038-4f42-4502-b4f8-cd57de3c0e86
drwxr-xr-x  3 rsiadmin rsiadmin 4096 Oct 18 17:27 dev-elasticsearch-data-ws-es-data-nodes-2-pvc-7446c823-b16e-4101-a309-d6c441e5841e
drwxr-xr-x  3 rsiadmin rsiadmin 4096 Oct 18 17:27 dev-elasticsearch-data-ws-es-master-nodes-1-pvc-52e9b807-ec8e-49bf-894f-9881140b24de
drwxr-xr-x  3 rsiadmin rsiadmin 4096 Oct 18 17:32 dev-elasticsearch-data-ws-es-master-nodes-1-pvc-f84aae88-ef12-4a1a-9628-8ddd6212723b

I changed the folder permissions to be 777 on the non-master nodes and the zookeeper pods finally restarted successfully. I had to apply this again as the kafka and entity-operator pods were subsequently deployed.

Note: I suspect that the difference in behaviour between elasticsearch and strimzi/kafka is that the strimzi image runs as a kafka user: https://github.com/strimzi/strimzi-kafka-operator/blob/master/docker-images/kafka/Dockerfile#L10-L12 This appears to be related: https://github.com/strimzi/strimzi-kafka-operator/issues/1720

[strigona-worksight]

@DallanQ Yes. I logged into each node and changed the folder permissions sudo chmod 777 *.

Switching from hostpath to something like Rook would be a better solution though.

[masus04]

@strigona-worksight And which folder would that be? Thx a lot