Use local persistent volumes for embedded journal, single master local journal, off-heap metastore and tiered storage

Is your feature request related to a problem? Please describe. Local persistent volumes is the preferred way for storage to persist across restarts of a pod (ref: https://kubernetes.io/docs/concepts/storage/volumes/#local). We should use persistent volumes for the following:

1. embedded journal volume for master pods
2. single master journal volume for master pod
3. off-heap metastore volume for master pods
4. tiered storage for worker pods

Describe the solution you’d like Helm charts should be extended to support local persistent volumes w/ resource limits (e.g., size of journal / tiered storage). Documentation w/ and w/o helm should be added as part of this ask. Note: hostPath volumes cannot be provisioned with resource limits and require root user and are hence not desired for persistent local storage.

1. embedded journal volume for master pods: restarts of the master pod w/o persistent volumes will lead to loosing metadata.

• The helm template statefulset.yaml should use persistentVolumeClaims instead of emptyDir volumes.
• journal-pvc.yaml should create masterCount claim resources
• journal-pv.yaml should be changed to use local persistent volume instead of hostPath by default

1. single master journal volume for master pod

• As part of pv for embedded journal we may choose to rename the claim from alluxio-pv-claim to alluxio-pv-claim-<i> where i=0 for a single master.

1. off-heap metastore volume for master pods

• Switch from emptyDir volume to localPV with limits

1. tiered storage for worker pods

• Switch from unbounded hostPath volume to localPV with limits

Describe alternatives you’ve considered Advanced end-user customizes the specs on their own

Urgency It would be painful w/o extensive knowledge of Alluxio to roll out raw specifications for these features to be production ready. This deployment will not be possible w/ helm (w/o this feature).

Additional context N/A

[jiacheliu3]

@madanadit After some research, I think the best practice for using local PV is by declaring a PVC on the Pod side (in fact I don’t find anybody directly declaring a local PV in the volumes). There are 2 ways to satisfy the PVC:

1. You can manually create local PV.

2. Use a provisioner to create local PVs for you like: https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/blob/master/docs/getting-started.md This is the static provisioner that K8s doc refers to.

In short, to use local PV, we need to support PVC for the declaration, and provide appropriate documentation for the user to provision the PVs for the PVCs.

[madanadit]

@cheyang FYI