[elasticsearch] Service type doesn't get set using helm chart available at elastic/elasticsearch but works with cloned chart.

Chart version: 7.1.1 Kubernetes version: Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0” Server Version: version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.4”

Kubernetes provider: E.g. GKE (Google Kubernetes Engine) Azure

Helm Version: Client: &version.Version{SemVer:“v2.12.0” Server: &version.Version{SemVer:“v2.13.1”

helm get release output

REVISION: 13
RELEASED: Mon Jun 24 08:24:21 2019
CHART: elasticsearch-7.1.1
USER-SUPPLIED VALUES:
clusterName: customers-integration-tests-elasticsearch
minimumMasterNodes: 1
replicas: 1
service:
  type: LoadBalancer

COMPUTED VALUES:
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
clusterHealthCheckParams: wait_for_status=green&timeout=1s
clusterName: customers-integration-tests-elasticsearch
esConfig: {}
esJavaOpts: -Xmx1g -Xms1g
esMajorVersion: 7
extraEnvs: []
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
fsGroup: 1000
fullnameOverride: ""
httpPort: 9200
image: docker.elastic.co/elasticsearch/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.1.1
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
initResources: {}
masterService: ""
maxUnavailable: 1
minimumMasterNodes: 1
nameOverride: ""
networkHost: 0.0.0.0
nodeAffinity: {}
nodeGroup: master
nodeSelector: {}
persistence:
  annotations: {}
  enabled: true
podAnnotations: {}
podManagementPolicy: Parallel
priorityClassName: ""
protocol: http
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 1000m
    memory: 2Gi
  requests:
    cpu: 100m
    memory: 2Gi
roles:
  data: "true"
  ingest: "true"
  master: "true"
secretMounts: []
service:
  type: LoadBalancer
sysctlVmMaxMapCount: 262144
terminationGracePeriod: 120
tolerations: []
transportPort: 9300
updateStrategy: RollingUpdate
volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

HOOKS:
---
# customers-integration-tests-elasticsearch-znpuz-test
apiVersion: v1
kind: Pod
metadata:
  name: "customers-integration-tests-elasticsearch-znpuz-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
  - name: "customers-integration-tests-elasticsearch-tdabf-test"
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.1.1"
    command:
      - "sh"
      - "-c"
      - |
        #!/usr/bin/env bash -e
        curl -XGET --fail 'customers-integration-tests-elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s'
  restartPolicy: Never
MANIFEST:

---
# Source: elasticsearch/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "customers-integration-tests-elasticsearch-master-pdb"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "customers-integration-tests-elasticsearch-master"
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: customers-integration-tests-elasticsearch-master
spec:
  selector:
    heritage: "Tiller"
    release: "customers-integration-tests-elasticsearch"
    chart: "elasticsearch-7.1.1"
    app: "customers-integration-tests-elasticsearch-master"
  ports:
  - name: http
    protocol: TCP
    port: 9200
  - name: transport
    protocol: TCP
    port: 9300
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: customers-integration-tests-elasticsearch-master-headless
  labels:
    heritage: "Tiller"
    release: "customers-integration-tests-elasticsearch"
    chart: "elasticsearch-7.1.1"
    app: "customers-integration-tests-elasticsearch-master"
  annotations:
    # Create endpoints also if the related pod isn't ready
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve
  selector:
    app: "customers-integration-tests-elasticsearch-master"
  ports:
  - name: http
    port: 9200
  - name: transport
    port: 9300
---
# Source: elasticsearch/templates/statefulset.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: customers-integration-tests-elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "customers-integration-tests-elasticsearch"
    chart: "elasticsearch-7.1.1"
    app: "customers-integration-tests-elasticsearch-master"
spec:
  serviceName: customers-integration-tests-elasticsearch-master-headless
  selector:
    matchLabels:
      app: "customers-integration-tests-elasticsearch-master"
  replicas: 1
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: customers-integration-tests-elasticsearch-master
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 30Gi

  template:
    metadata:
      name: "customers-integration-tests-elasticsearch-master"
      labels:
        heritage: "Tiller"
        release: "customers-integration-tests-elasticsearch"
        chart: "elasticsearch-7.1.1"
        app: "customers-integration-tests-elasticsearch-master"
      annotations:

    spec:
      securityContext:
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "customers-integration-tests-elasticsearch-master"
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 120
      volumes:
      initContainers:
      - name: configure-sysctl
        securityContext:
          runAsUser: 0
          privileged: true
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.1.1"
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        resources:
          {}

      containers:
      - name: "elasticsearch"
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.1.1"
        imagePullPolicy: "IfNotPresent"
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file

                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
                }

                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    http "/"
                else
                    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                        exit 1
                    fi
                fi
        ports:
        - name: http
          containerPort: 9200
        - name: transport
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 2Gi

        env:
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: cluster.initial_master_nodes
            value: "customers-integration-tests-elasticsearch-master-0,"
          - name: discovery.seed_hosts
            value: "customers-integration-tests-elasticsearch-master-headless"
          - name: cluster.name
            value: "customers-integration-tests-elasticsearch"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
            value: "-Xmx1g -Xms1g"
          - name: node.data
            value: "true"
          - name: node.ingest
            value: "true"
          - name: node.master
            value: "true"
        volumeMounts:
          - name: "customers-integration-tests-elasticsearch-master"
            mountPath: /usr/share/elasticsearch/data
      # This sidecar will prevent slow master re-election
      # https://github.com/elastic/helm-charts/issues/63
      - name: elasticsearch-master-graceful-termination-handler
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.1.1"
        imagePullPolicy: "IfNotPresent"
        command:
        - "sh"
        - -c
        - |
          #!/usr/bin/env bash
          set -eo pipefail

          http () {
              local path="${1}"
              if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
              else
                BASIC_AUTH=''
              fi
              curl -XGET -s -k --fail ${BASIC_AUTH} http://customers-integration-tests-elasticsearch-master:9200${path}
          }

          cleanup () {
            while true ; do
              local master="$(http "/_cat/master?h=node")"
              if [[ $master == "customers-integration-tests-elasticsearch-master"* && $master != "${NODE_NAME}" ]]; then
                echo "This node is not master."
                break
              fi
              echo "This node is still master, waiting gracefully for it to step down"
              sleep 1
            done

            exit 0
          }

          trap cleanup SIGTERM

          sleep infinity &
          wait $!
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name

Describe the bug: When using the helm chart available at elastic/elasticsearch the service.type doesn’t get set. Command used: helm upgrade --install --force --set clusterName=customers-integration-tests-elasticsearch,replicas=1,minimumMasterNodes=1,service.type=LoadBalancer customers-integration-tests-elasticsearch elastic/elasticsearch

Output from command:

Release "customers-integration-tests-elasticsearch" has been upgraded. Happy Helming!
LAST DEPLOYED: Mon Jun 24 08:37:07 2019
NAMESPACE: vinci
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                                                READY  STATUS   RESTARTS  AGE
customers-integration-tests-elasticsearch-master-0  3/3    Running  0         13m

==> v1/Service
NAME                                                       TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
customers-integration-tests-elasticsearch-master           ClusterIP  10.0.61.70  <none>       9200/TCP,9300/TCP  14m
customers-integration-tests-elasticsearch-master-headless  ClusterIP  None        <none>       9200/TCP,9300/TCP  54m

==> v1beta1/PodDisruptionBudget
NAME                                                  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
customers-integration-tests-elasticsearch-master-pdb  N/A            1                1                    54m

==> v1beta1/StatefulSet
NAME                                              READY  AGE
customers-integration-tests-elasticsearch-master  1/1    54m


NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=vinci -l app=customers-integration-tests-elasticsearch-master -w
2. Test cluster health using Helm test.
  $ helm test customers-integration-tests-elasticsearch

When using the same chart but cloned locally with exactly the same values it does get set. Command used: (in chart folder) helm upgrade --install --force --set clusterName=customers-integration-tests-elasticsearch,replicas=1,minimumMasterNodes=1,service.type=LoadBalancer customers-integration-tests-elasticsearch .

Output from command:

Release "customers-integration-tests-elasticsearch" has been upgraded. Happy Helming!
LAST DEPLOYED: Mon Jun 24 08:37:51 2019
NAMESPACE: vinci
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                                                READY  STATUS       RESTARTS  AGE
customers-integration-tests-elasticsearch-master-0  3/3    Terminating  0         13m

==> v1/Service
NAME                                                       TYPE          CLUSTER-IP  EXTERNAL-IP  PORT(S)                        AGE
customers-integration-tests-elasticsearch-master           LoadBalancer  10.0.61.70  <pending>    9200:30184/TCP,9300:32622/TCP  15m
customers-integration-tests-elasticsearch-master-headless  ClusterIP     None        <none>       9200/TCP,9300/TCP              54m

==> v1beta1/PodDisruptionBudget
NAME                                                  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
customers-integration-tests-elasticsearch-master-pdb  N/A            1                0                    54m

==> v1beta1/StatefulSet
NAME                                              READY  AGE
customers-integration-tests-elasticsearch-master  1/1    54m`

**Steps to reproduce:**

1. Deploy helm chart elastic/elasticsearch and set value `service
.type=LoadBalancer

Steps to reproduce:

1. Deploy chart form elastic/elasticsearch and use --set service.type=LoadBalancer

Expected behavior: Service type should be set to LoadBalancer

Any additional context:

This is really wierd behaviour and I haven’t seen anything similar previously. Looking through the helm chart I can’t understand the reason for these different behaviours between using the repository and local versions.

I’m not a helm master so maybe I’m missing something obvious here. If anyone has any ideas they would be helpful.

For now I will include the chart in our repository for our deployments.

[Crazybus]

I am also interested in changing service type, @Crazybus any update about next release?

I think you already saw it, but this was released yesterday in the 7.2.0 release. https://github.com/elastic/helm-charts/releases/tag/7.2.0

[Crazybus]

When the next release is planned? 😉

Will be later this week. Most likely on Wednesday.

Yikes, I’m so used to always having master as the latest release that I din’t even consider that.

Couple of reasons that makes this tricky:

1. We release all of the charts together at the same time with the same version in sync with the stack. In theory we could have a script that dynamically adds an extra version number on the end like 7.1.1-1 and release on every change to master. However:
2. We don’t currently have a merge bot setup. This means that pull requests are not tested against a rebased version of master. So a passing pull request build might fail in master once it has actually been re-based.
3. Because the charts are designed to work together there are sometimes multiple PRs that might end up depending on each other.

All of these things are very fixable and automatable but require time and effort. So this is the situation for now, but one day we might be able to move to a “release on every merge to master” model.