Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Helm chart: How to enable HTTPS connection

See original GitHub issue

Apache Airflow version: 2.0.2 Helm chart version: 1.0.0 Kubernetes version: 1.20

What happened: I have tried to enable HTTPS connection, however when I enable it liveness and readiness probes are failing

▶ kubectl -n airflow describe pod/airflow-webserver-564fcf9c5c-jqtt4

Name:         airflow-webserver-564fcf9c5c-jqtt4
Namespace:    airflow
Priority:     0
Node:         ip-10-10-13-31.eu-central-1.compute.internal/10.10.13.31
Start Time:   Sun, 23 May 2021 11:37:08 +0200
Labels:       component=webserver
              pod-template-hash=564fcf9c5c
              release=airflow
              tier=airflow
Annotations:  checksum/airflow-config: 4e5f26f48f157218dcaf8eddac95e912aa7a47a4eb75b04697bb3701fd550340
              checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
              checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
              checksum/metadata-secret: 765446a9def21895960ebc0df295399c35d3467cdd84be6ef8bc65c18ca0f7e5
              checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
              checksum/webserver-config: 4a2281a4e3ed0cc5e89f07aba3c1bb314ea51c17cb5d2b41e9b045054a6b5c72
              kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.10.13.56
IPs:
  IP:           10.10.13.56
Controlled By:  ReplicaSet/airflow-webserver-564fcf9c5c
Init Containers:
  wait-for-airflow-migrations:
    Container ID:  docker://da6e1a36ac496982bc6d9058e1d94791cf99bf7d70dbbe10be2f2bbe0446eae8
    Image:         apache/airflow:2.0.2
    Image ID:      docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
    Port:          <none>
    Host Port:     <none>
    Args:
      python
      -c
      import airflow
      import logging
      import os
      import time

      from alembic.config import Config
      from alembic.runtime.migration import MigrationContext
      from alembic.script import ScriptDirectory

      from airflow import settings

      package_dir = os.path.abspath(os.path.dirname(airflow.__file__))
      directory = os.path.join(package_dir, 'migrations')
      config = Config(os.path.join(package_dir, 'alembic.ini'))
      config.set_main_option('script_location', directory)
      config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN.replace('%', '%%'))
      script_ = ScriptDirectory.from_config(config)

      timeout=60

      with settings.engine.connect() as connection:
          context = MigrationContext.configure(connection)
          ticker = 0
          while True:
              source_heads = set(script_.get_heads())

              db_heads = set(context.get_current_heads())
              if source_heads == db_heads:
                  break

              if ticker >= timeout:
                  raise TimeoutError("There are still unapplied migrations after {} seconds.".format(ticker))
              ticker += 1
              time.sleep(1)
              logging.info('Waiting for migrations... %s second(s)', ticker)

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 23 May 2021 11:37:11 +0200
      Finished:     Sun, 23 May 2021 11:37:24 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      AIRFLOW__CORE__FERNET_KEY:        <set to the key 'fernet-key' in secret 'airflow-fernet-key'>         Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:          <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AWS_DEFAULT_REGION:               eu-central-1
      AWS_REGION:                       eu-central-1
      AWS_ROLE_ARN:                     arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
      AWS_WEB_IDENTITY_TOKEN_FILE:      /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb (ro)
Containers:
  webserver:
    Container ID:  docker://2b8ee8a99ebd088b545685b3d1bc290adb0a7675e4935322a9549a152339a249
    Image:         apache/airflow:2.0.2
    Image ID:      docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      bash
      -c
      exec airflow webserver
    State:          Running
      Started:      Sun, 23 May 2021 11:39:48 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 23 May 2021 11:37:26 +0200
      Finished:     Sun, 23 May 2021 11:39:48 +0200
    Ready:          False
    Restart Count:  1
    Liveness:       http-get http://:8080/airflow/health delay=15s timeout=30s period=5s #success=1 #failure=20
    Readiness:      http-get http://:8080/airflow/health delay=15s timeout=30s period=5s #success=1 #failure=20
    Environment:
      AIRFLOW__CORE__FERNET_KEY:        <set to the key 'fernet-key' in secret 'airflow-fernet-key'>         Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:          <set to the key 'connection' in secret 'airflow-postgres-password'>  Optional: false
      AWS_DEFAULT_REGION:               eu-central-1
      AWS_REGION:                       eu-central-1
      AWS_ROLE_ARN:                     arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
      AWS_WEB_IDENTITY_TOKEN_FILE:      /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /opt/airflow/dags from dags (rw)
      /opt/airflow/logs from logs (rw)
      /opt/airflow/secrets/ssl from airflow-ssl (ro)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-airflow-config
    Optional:  false
  dags:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  airflow-dags
    ReadOnly:   false
  logs:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  airflow-logs
    ReadOnly:   false
  airflow-ssl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-ssl
    Optional:    false
  airflow-token-9cqzb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-token-9cqzb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  namespace=airflow
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m14s                  default-scheduler  Successfully assigned airflow/airflow-webserver-564fcf9c5c-jqtt4 to ip-10-10-13-31.eu-central-1.compute.internal
  Normal   Pulled     4m11s                  kubelet            Container image "apache/airflow:2.0.2" already present on machine
  Normal   Created    4m11s                  kubelet            Created container wait-for-airflow-migrations
  Normal   Started    4m11s                  kubelet            Started container wait-for-airflow-migrations
  Normal   Pulled     3m57s                  kubelet            Container image "apache/airflow:2.0.2" already present on machine
  Normal   Created    3m57s                  kubelet            Created container webserver
  Normal   Started    3m56s                  kubelet            Started container webserver
  Warning  Unhealthy  3m37s                  kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp 10.10.13.56:8080: connect: connection refused
  Warning  Unhealthy  3m36s (x2 over 3m41s)  kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp 10.10.13.56:8080: connect: connection refused
  Warning  Unhealthy  3m2s                   kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  3m1s                   kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:37846->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  3m1s                   kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38016->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  3m1s                   kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38046->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  3m1s                   kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38048->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  2m57s                  kubelet            Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38056->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  2m56s                  kubelet            Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38060->10.10.13.56:8080: read: connection reset by peer
  Warning  Unhealthy  2m32s (x9 over 2m52s)  kubelet            (combined from similar events): Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38192->10.10.13.56:8080: read: connection reset by peer

How to reproduce it: Helm chart myconf.yaml looks like this

executor: KubernetesExecutor
fernetKey: "XXXXXXXXXX"

webserver:
  base_url: "https://k8s.foobaa.com/airflow"
  web_server_ssl_cert: "/opt/airflow/secrets/ssl/tls.crt"
  web_server_ssl_key: "/opt/airflow/secrets/ssl/tls.key"

webserver:
  service:
    type: NodePort
  extraVolumeMounts:
    - name: airflow-ssl
      mountPath: /opt/airflow/secrets/ssl
      readOnly: true
  extraVolumes:
    - name: airflow-ssl
      secret:
        secretName: airflow-ssl

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:20 (14 by maintainers)

github_iconTop GitHub Comments

3reactions
andormarkuscommented, May 25, 2021

@Anirudhaagrawal During my initial comment I was’n fully familiar with the helm chart values file. potiuk and mik-laj recommended: we should not use HTTPS inside kubernetes, it will fail on the liveness and readiness probes.

Recommended traffic flow looks like this: internet --> HTTPS --> K8s ingress controller --> HTTP --> Airflow webserver

my configuration looks like this (AWS EKS with aws-load-balancer-controller):

config:
  webserver:
    base_url: https://k8s.foobaa.com/airflow

ingress:
  enabled: true
  web:
    precedingPaths:
      - path: "/*"
        serviceName: "ssl-redirect"
        servicePort: "use-annotation"
    path: "/airflow/*"

    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600

      alb.ingress.kubernetes.io/certificate-arn: ${aws_acm_certificate_arn}
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
2reactions
kaxilcommented, May 25, 2021

I am going to close this ticket for now with https://github.com/apache/airflow/issues/16010#issuecomment-847929472 as the correct solution. For other issues #16013 #16020 – We will take a look and comment on those specific tickets

Read more comments on GitHub >

github_iconTop Results From Across the Web

How can I create a HELM https service? - Stack Overflow
I am wanting to use kubernetes cluster root certificate as described here: How to access a kubernetes service through https?
Read more >
Enabling TLS on the Logstream-leader Helm Chart - Cribl
Use the config.healthScheme Helm chart option to set the liveness and readiness checks to use HTTPS. Use the extraConfigmapMounts Helm chart ...
Read more >
How to enable the https instead of http in sonarqube-lts helm ...
Currently sonarqube is up and running by using port-forward( http://localhost:8080 ) Now i want to enable the https instead of http. so i...
Read more >
Enable HTTPS for your application on Kubernetes in 5 steps ...
Step 1: Setup NGINX Ingress Controller. Assuming helm is already installed and kubectl is configured correctly on your laptop, let's setup nginx ...
Read more >
Enabling https for helm deployed dremio
It deploys and works fine but I want to use https to access the app ... and the helm chart in https://github.com/dremio/dremio-cloud-tools.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

CLI 'kubernetes cleanup-pods' fails on invalid label key

Next page

Masking passwords with empty connection passwords make some logs unreadable in 2.1.0