[APM-SERVER] It is not possible to configure HTTPS on the pod

Chart version: 7.6.1

Kubernetes version: 1.17

Kubernetes provider: E.g. GKE (Google Kubernetes Engine) GKE, Kind

Helm Version: 3.0.2

helm get release output

USER-SUPPLIED VALUES:
apmConfig:
  apm-server.yml: |
    logging.level: debug
    apm-server:
      host: "0.0.0.0:8200"
      rum.enabled: true
      ilm.enabled: true
      secret_token: ${APM_TOKEN}
# HERE is the issue if it is true the livenessProbe and readinessProbe fails
      ssl.enabled: true
      ssl.key: /usr/share/apm-server/config/certs/tls.key
      ssl.certificate: /usr/share/apm-server/config/certs/tls.crt
    queue: {}
    monitoring.elasticsearch.ssl.verification_mode: none
    setup.kibana:
      host: ${KIBANA_PROTO}://${KIBANA_HOST}:${KIBANA_PORT}
      protocol: ${KIBANA_PROTO}
      username: ${KIBANA_USERNAME}
      password: ${KIBANA_PASSWORD}
    output.elasticsearch:
      enabled: true
      hosts: ["${ELASTICSEARCH_PROTO}://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.verification_mode: none
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: user
      name: es-creds
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: es-creds
- name: ELASTICSEARCH_PROTO
  valueFrom:
    secretKeyRef:
      key: protocol
      name: es-creds
- name: ELASTICSEARCH_HOST
  valueFrom:
    secretKeyRef:
      key: host
      name: es-creds
- name: ELASTICSEARCH_PORT
  valueFrom:
    secretKeyRef:
      key: port
      name: es-creds
- name: NODE_NAME
  valueFrom:
    fieldRef:
      fieldPath: spec.nodeName
- name: KIBANA_HOST
  valueFrom:
    secretKeyRef:
      key: host
      name: kibana-creds
- name: KIBANA_PROTO
  valueFrom:
    secretKeyRef:
      key: protocol
      name: kibana-creds
- name: KIBANA_PORT
  valueFrom:
    secretKeyRef:
      key: port
      name: kibana-creds
- name: KIBANA_USERNAME
  valueFrom:
    secretKeyRef:
      key: user
      name: kibana-creds
- name: KIBANA_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: kibana-creds
- name: APM_TOKEN
  valueFrom:
    secretKeyRef:
      key: token
      name: apm-creds
extraVolumeMounts:
- mountPath: /usr/share/apm-server/config/certs
  name: default-tls
extraVolumes:
- name: default-tls
  secret:
    defaultMode: 420
    secretName: default-tls
image: elastic/apm-server
imagePullPolicy: Always
imageTag: 7.6.1
ingress:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
    ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  enabled: true
  hosts:
  - apm.IP.ip.es.io
  path: /
  tls:
  - hosts:
    - apm.IP.ip.es.io
    secretName: apm-ingress-tls
replicas: 2
service:
  annotations: {}
  port: "8200"
  targetPort: "8200"
  type: ClusterIP

Describe the bug:

If you configure the APM Server to use TLS the pod is killed by the livenessProbe check. livenessProbe and readinessProbe only check HTTP so they fail if you configure HTTPS

Steps to reproduce:

1. Install the Helm chart with the provided configuration
2. wait a couple of mins
3. check the pod logs to confirm the pod is killed

Expected behavior:

The APM server goes up and running, a way to configure the livenessProbe and readinessProbe to use HTTPS would resolve the issue.

Provide logs and/or server output (if relevant):

2020-03-19T12:21:42.773Z	DEBUG	[monitoring]	memqueue/ackloop.go:128	ackloop: return ack to broker loop:1
2020-03-19T12:21:42.774Z	DEBUG	[monitoring]	memqueue/ackloop.go:131	ackloop:  done send ack
2020-03-19T12:21:52.720Z	DEBUG	[service]	service/service.go:53	Received sigterm/sigint, stopping
2020-03-19T12:21:52.720Z	INFO	[beater]	beater/beater.go:230	stopping apm-server... waiting maximum of 5 seconds for queues to drain
2020-03-19T12:21:52.720Z	INFO	[beater]	beater/http.go:103	Stop listening on: [::]:8200
2020-03-19T12:21:52.721Z	INFO	[beater]	beater/server.go:88	Server stopped
2020-03-19T12:21:52.721Z	DEBUG	[publisher]	pipeline/client.go:162	client: closing acker

[mfinelli]

Another solution is to just override the liveness and readiness probes to use an HTTPS check:

livenessProbe:
  httpGet:
    path: /
    port: http
    scheme: HTTPS
  initialDelaySeconds: 30
  failureThreshold: 3
  periodSeconds: 10
  timeoutSeconds: 5
readinessProbe:
  httpGet:
    path: /
    port: http
    scheme: HTTPS
  initialDelaySeconds: 30
  failureThreshold: 3
  periodSeconds: 10
  timeoutSeconds: 5

[kuisathaverat]

overwriting the probes and using the schema HTTPS resolves the issue, see https://github.com/elastic/helm-charts/issues/527#issuecomment-856830956