How to resize ES storage volume but keeping old data

Hi all, I’ll need a help with resizing a storage for elasticsearch. We have deployed elasticserach with helm but we have an issue with resizing the storage. This are the resources that we deployed with helm:

> pod/sandbox-elasticsearch-test2-0                             1/1     Running   0          63m
> pod/sandbox-elasticsearch-test2-1                             1/1     Running   0          63m
> 
> service/sandbox-elasticsearch-test2                    ClusterIP   172.20.15.222    <none>        9200/TCP,9300/TCP            63m
> service/sandbox-elasticsearch-test2-headless           ClusterIP   None             <none>        9200/TCP,9300/TCP            63m
> statefulset.apps/sandbox-elasticsearch-test2                     2/2     63m

We tried to resize the storage by changing nodeGroup and masterService and we ended with new nodes but old one are deleted, only what is left is old storage disks :

sandbox-elasticsearch-test-sandbox-elasticsearch-test-0                                    Bound    pvc-f45fe905-7f09-43aa-b4a5-5ef8124ad69e   120Gi      RWO            gp2-csi        18h
sandbox-elasticsearch-test-sandbox-elasticsearch-test-1                                    Bound    pvc-5ed567f6-0aa4-4ebd-8162-42f7fa27ee0d   120Gi      RWO            gp2-csi        18h
sandbox-elasticsearch-test2-sandbox-elasticsearch-test2-0                                  Bound    pvc-46a87003-3c90-453c-a3dc-de65877d60b7   130Gi      RWO            gp2-csi        64m
sandbox-elasticsearch-test2-sandbox-elasticsearch-test2-1                                  Bound    pvc-334cfc32-a376-469a-ae9d-e7e0e2cf3c7a   130Gi      RWO 

I think data is still on old disks but how to migrate old data to new nodes ? After the attempt to resize the storage we ended up with a brand new cluster and old data are lost .

[jmlrt]

Hi @milijander88, The good news is that PVC are never deleted by this chart so changing back nodeGroup and masterService to previous values should be enough to re-use the disks with datas.

As for the procedure to resize disk, changing volumeClaimTemplate.resources.requests.storage value would fail with Updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden error because this field is immutable at Kubernetes level. That’s why even @ebuildy procedure would fail at step 1.

The recommended workaround for Elasticsearch chart is using a new chart release, with different storage size to replace the existing one.:

1. Install a new deployment of the helm chart, with nodeGroup set to a unique value, such as data-1 and the appropriate PVC claim size.
2. Ensure that these new ES nodes have joined the cluster.
3. Trigger Elasticsearch to move all data off of the data nodes. This can be achieved by using Cluster-level shard allocation filtering, using the IP’s of the 3 original data nodes.
4. Once all data has been migrated off of the original data nodes, then delete that helm deployment.

(source: https://github.com/elastic/helm-charts/issues/647#issuecomment-636778450)

This is related to #772, #893 and #647.

[Xat59]

@jmlr I am now able to resize the pvc using a second release in parallel, wait for the data sync, then removing the original.

But, doing that, it also renames the elasticsearch service regarding $clusterName-$nodeGroup . And this make the service unavailable because it has been renamed.

Any idea how to do it properly ?

EDIT : actually I’ve resolve it by creating the multi chart release, at least one for the master and one for the data, as described here : https://github.com/elastic/helm-charts/tree/master/elasticsearch/examples/multi