rfc(template): remove custom Jinja templating

As Reana uses Helm to install Traefik, I was thinking why not use Helm for back-end kubernetes instead of using custom templates with jinja. Since Helm is an industry standard for reusable Kubernetes packages, it can open up opportunities for independent deployment without a reana-cluster and allow the use of packages such as stable/postgresql.

I look forward to contributing more 🥰

[pcasteran]

Indeed Helm’s current model require special care when used on a Kubernetes cluster with Role-based access control (RBAC) enabled, which is the case for example on Google’s managed clusters (GKE). Following Helm’s official documentation on RBAC, some Kubernetes resources (service account, cluster role and binding, …) are required and must be created before deploying Tiller.

If you are interested, I can submit the configuration required for Helm to deploy Reana in a RBAC-enabled cluster.

[shikanime]

Hello @everyone

Of course, to start with, I started the project using Helm since it is the tool I used the most. Since September, I have moved all my work to Kustomize. I will objectively state below the reason and how Kustomize solved my problems and why i now exclusively use Kustomize.

Helm is one of the first package managers for Kubernetes and is an industry standard for the moment after the vanilla Kubernetes YAML manifest. Like Ksonnet, Helm uses a programmatic model instead of a declarative model to produce Kubernetes manifests.

On the other hand, Kustomize is template-free and declarative to extend the manifests and is natively supported by Kubernetes.

Organization

Helm 2

Helm uses the Chart.yaml file as an entry point that defines the version, name, description… of the Chart. And values.yaml is where the parameters are defined to be passed down to the template.

./
├── Chart.yaml
├── values.yaml
├── requirements.lock
├── requirements.lock
├── files/
│   └── ...
├── charts/
│   ├── postgresql.tar.gz
│   ├── rabbitmq.tar.gz
│   ├── reana-ui
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   ├── requirements.lock
│   │   ├── requirements.lock
│   │   └── templates/
│   │       ├── *.tpl
│   │       ├── *.yaml
│   │       └── ...
│   └── ...
└── templates/
    ├── *.tpl
    ├── *.yaml
    └── ...

And it allows you to conditionally import nested subcharts and third-party charts.

./requirements.yaml:

dependencies:
  - name: postgreql
    version: 6.3.9
    repository: https://kubernetes-charts.storage.googleapis.com/
  - name: rabbitmq
    version: 6.7.4
    repository: https://kubernetes-charts.storage.googleapis.com/
  - name: reana-ui
    version: 0.1.0
    condition: reanaui.enabled
    repository: file://subcharts/reanaui

Kustomize

A Kustomize project structure is more minimalist and is unopinionated, using kustomization.yaml as the entry point for the package, structurally similar to Terraform. Kustomize allows us to model our structure according to our needs.

Environment oriented structure :

./
├── base/
│   ├── kustomization.yaml
│   │── *.yaml
│   └── ...
├── dev/
│   ├── base/
│   │   ├── kustomization.yaml
│   │   ├── *.yaml
│   │   └── ...
│   └── minikube/
│       ├── kustomization.yaml
│       ├── *.yaml
│       └── ...
└── prod/
    ├── base/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    └── google-cloud/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    └── cern/
        ├── kustomization.yaml
        ├── *.yaml
        └── ...

Overlays:

./
├── base/
│   ├── kustomization.yaml
│   │── *.yaml
│   └── ...
└── overlays/
    ├── cephfs/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    ├── minikube/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    ├── ui/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    └── traefik/
        ├── kustomization.yaml
        ├── *.yaml
        └── ...

Kustomize in constrast is more declarative, kustomization.yaml hold every Kubernetes manifests in the resources list and other package imports in the bases using Hashicorp go getter format. And have several other utility such as:

• commonLabel to put a common every children resources;
• *Generator to create secret or configmap directly from litteral or from a file path.

./base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  app.kubernetes.io/part-of: reana
  app.kubernetes.io/managed-by: kustomize
bases:
  - postgresql
  - redis
  - rabbitmq
  - reana-server
  - reana-ui
  - reana-wdb
  - reana-workflow-controller
# ...

./base/postgresql/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  app.kubernetes.io/name: postgresql
  app.kubernetes.io/version: "11.5.0"
  app.kubernetes.io/component: database
resources:
  - headless-svc.yaml
  - statefulset.yaml
  - svc.yaml
secretGenerator:
  - literals:
      - postgresql-password=reana
    name: postgresql
# ...

Configuration

Helm

One of the main restrictions of templating is that it’s not extensible by default. Historically, templating is good for ahead known configuration but in case of infrastructure deployment that’s not the case, sometime we need to pass certain environment variable or volume based on various variables. Which lead to template a configuration to be configured on the Chart level.

./charts/reanaserver/templates/deployment.yaml

# ...
          env:
            {{- with .Values.hostName }}
            - name: REANA_URL
              value: {{ . }}
            {{- end }}
{{ include "reanaserver.env" . | indent 12 }}
{{- if .Values.extraEnv }}
{{ tpl (toYaml .Values.extraEnv) $ | indent 12 }}
{{- end }}
          ports:
            - name: rest-api
              containerPort: 5000
        - name: scheduler
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command: ["flask", "start-scheduler"]
          resources:
{{- toYaml .Values.scheduler.resources | indent 12 }}
          {{- if .Values.securityContext.enabled }}
          securityContext:
            runAsUser: {{ .Values.securityContext.runAsUser }}
          {{- end }}
          env:
{{ include "reanaserver.env" . | indent 12 }}
{{- if .Values.extraEnv }}
{{ tpl (toYaml .Values.extraEnv) $ | indent 12 }}
{{- end }}
# ...

./values.yaml

reanaserver:
  extraVolumeMounts:
    - name: reana-shared-volume
      mountPath: "/var/reana"
  extraVolumes:
    - name: reana-shared-volume
      persistentVolumeClaim:
        claimName: manila-cephfs-pvc
        readOnly: false
  extraEnv:
    - name: SHARED_VOLUME_PATH
      value: reana-shared-volume
    - name: REANA_STORAGE_BACKEND
      value: "cephfs"

Kustomize

This is where Kustomize shine, it use template-free Kubernetes manifest and extend it by applying the superposition.

./base/postgresql/deploy.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reana-server
spec:
  selector:
    matchLabels: &containerLabels
      app: reana-server
  template:
    metadata:
      labels: *containerLabels
    spec:
      containers:
        - env:
            - name: REANA_DEPLOYMENT_TYPE
              value: development
            - name: SHARED_VOLUME_PATH
              value: /var/reana
            - name: REANA_DB_HOST
              value: $(POSTGRESQL_NAME)
            - name: REANA_DB_PORT
              value: "5432"
            # ...
          name: server
          volumeMounts:
            - mountPath: /var/reana
              name: reana-shared-volume
          # ...

./prod/cern/postgresql/deploy-patch.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reana-server
spec:
  template:
    spec:
      containers:
        - name: server
          env:
            - name: REANA_DEPLOYMENT_TYPE
              value: production
            - name: REANA_STORAGE_BACKEND
              value: "cephfs"
    - name: reana-shared-volume
      mountPath: "/var/reana"
  volumes:
    - name: reana-shared-volume
      persistentVolumeClaim:
        claimName: manila-cephfs-pvc
        readOnly: false

Deployment and security

Helm

Firstly Helm 2 manages its releases via the tiller, which is a security issue by itself in the same way as the RBAC security issues of the Kubenetes dashboard. Partially solved in the Helm 3 by using an opaque configmap instead of Tiller.

helm install --name reana-test ./chart

Kustomize

In contrast, Kustomize is natively supported by Kubernetes and is easier to deploy by simply using the following command on a directory that contains a file kustomization.yaml:

kubectl apply/create/delete -k ./manifests

Apply and prune all deleted resources:

kubectl apply --prune -k ./manifests

Conclution

At each step Kustomize feels more Kubernetes. With or without Kustomize, all the manifests written are valid vanilla Kubernetes manifest and are extensible by default without requiring to fork or create a pull request.

Helm is harsh and feels magical where Kustomize is declarative following the Kubernetes philosophy. The resources declared in Kustomize are ordered by priority such as CRD then configmap then… (https://github.com/helm/helm/issues/5871). Helm templates (*.tpl) are also another pain point point and cannot be shared or have strange behaviors between charts (https://github.com/helm/helm/issues/3920)