Better support exposing the Alluxio service to external the K8s cluster

Is your feature request related to a problem? Please describe. It’s becoming more common that users want to host the Alluxio service on K8s while some external applications need to access the Alluxio cluster from outside the K8s cluster.

In the current state, the users need to:

1. Change the master K8s services to be accessible from outside the K8s cluster. This typically changes the service to NodePort or Ingress.
2. Somehow expose the worker pods to the external. This is much harder than 1 because worker pods are dynamic and do not have associated Services. One way is to use hostNetwork=true for all workers and clients will then talk to worker nodes.

Describe the solution you’d like We need one solution for:

1. Enabling master pods to be accessible from outside
2. Enabling worker pods to be accessible from outside
3. Ideally use only one switch to control all

The biggest challenge is the worker pods. Using a combination of StatefulSet deployed workers + externalTrafficPolicy Service can be a solution. The Service maps to the worker pod by name, which becomes deterministic because workers are now deployed with StatefulSet.

apiVersion: v1
kind: Service
metadata:
  name: worker-0
spec:
  type: NodePort
  externalTrafficPolicy: Local
  selector:
    statefulset.kubernetes.io/pod-name: worker-0
  ports:
  - protocol: TCP
    port: 19998
    targetPort: 19998

The worker pods now need anti affinity defined, so no two worker pods appear on one node.

The master pods can be exposed similarly.

Describe alternatives you’ve considered Use hostNetwork to deploy all master and worker pods and access the Alluxio pods by node IP. This is the cleanest way as of Alluxio v2.8. The challenge is hostNetwork requires admin privileges and may even incur port collision with other services.

Urgency MEDIUM. There are existing use cases for this setup.

Additional context Add any other context or screenshots about the feature request here.

[nirav-chotai]

Solution should be independent whether hostNetwork is enabled or not.

Init container for workers should collect metadata and use an init script to talk to master and register themselves.

          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace

and

            - register-worker
            - --ip
            - $(POD_IP)
            - --k8s-namespace
            - $(POD_NAMESPACE)

Something like above.

This will work regardless I enabled hostNetwork or not.

      hostNetwork: {{ $hostNetwork }}
      hostPID: {{ $hostPID }}
      dnsPolicy: {{ .Values.worker.dnsPolicy | default ($hostNetwork | ternary "ClusterFirstWithHostNet" "ClusterFirst") }}

[jiacheliu3]

No the masters don’t have that equivalent because we use Service to handle the name resolution. Clients talk to services so no need to know the pod names. But yea we currently don’t have a uniformed definition of what hostnames map to which use cases (internal/external the k8s cluster etc). The existing configs are more on demand. If there’s a chance to unify all those, I’m totally in 😃