Unable to connect to redis in aws that requires encrypted data in flight

Apache Airflow version: 2.0.1

Environment: AWS

• Cloud provider or hardware configuration: AWS using redis configured to encrypt data in flight
• OS (e.g. from /etc/os-release): docker image based on apache/airflow:2.0.1 running in ECS

What happened:

When deploying the flower service, clicking on the broker tab of the ui would just hang. Nothing got logged.

What you expected to happen: I expected to see broker status

How to reproduce it: Point your airfow cluster at a redis cluster that requires TLS

Anything else we need to know:

This bug report is as much a question as it is a bug.

Yesterday I spent a bunch of time debugging why flower in aws would hang every time the broker dashboard was accessed.

The answer: calls to the redis backend would hang indefinitely because redis was configured to encrypt data in flight.

Without the tls negotiation on handshake, any call (ping, get, put) would block forever.

Nothing interesting showed in the cloudwatch logs and at first i thought this was just a tcp connectivity issue.

Evenutally i got frustrated with trying to debug it remotely so i hacked up a docker compose that let me run things locally, but connect to my aws infra.

At that point I identified the problem and confirmed it with a quick program using the python redis module directly.

Next I read your docs for configuring airflow. The ssl options are not clearly documented, and seem to be there for people who are using self singed certs so that they can get their trust store/peers working together.

In aws, the certs are all signed by a ‘real’ CA that’s already in the default trust store. So there’s no reason to mess with keys/cacerts/etc.

I ended up reading thru the celery source looking for how it worked when I spotted this: ` 15:27 $ rg ‘rediss://’ celery/backends/redis.py 71:is redis://. A Redis SSL connection URL should use the scheme rediss://. 75:A rediss:// URL must have parameter ssl_cert_reqs and this must be set to \

Read up more on the rediss:// prefix on the celery docs and it seemed to be what I was after:

diff --git a/airflow/config_templates/default_celery.py b/airflow/config_templates/default_celery.py index 7231db54f…6f460e147 100644 — a/airflow/config_templates/default_celery.py +++ b/airflow/config_templates/default_celery.py @@ -65,7 +65,7 @@ try: ‘ca_certs’: conf.get(‘celery’, ‘SSL_CACERT’), ‘cert_reqs’: ssl.CERT_REQUIRED, }

•    elif 'redis://' in broker_url:
  

•    elif 'redis://' in broker_url or 'rediss://' in broker_url:
         broker_use_ssl = {
             'ssl_keyfile': conf.get('celery', 'SSL_KEY'),
             'ssl_certfile': conf.get('celery', 'SSL_CERT'),
  

`

So, I created the trivial patch to airflow above and it worked as desired.

My question is, is this really needed? If there is a better way for me to use the existing config options, please educated me.

At the end of the day, I have docker images in ecs that are operational, I just don’t like maintaining one off patches that muddy up the upgrade path.

Celery seems to see the string literal ‘rediss://’ in the url and it just ‘does the right thing’. So I think this is needed for folks who are using redis in aws who also have the encrypt in transit boxes checked on their redis configurations.

So, again to be clear, this isn’t the same as configuring ssl keys, cacerts and all that. This is literally just asking celery to negotiate a tls connection to the broker via the usual mechanisms.

You will also want this in your airflow.cfg:

[celery] ssl_active = True

[jstelzer]

In my case I’m using AIRFLOW__CELERY__BROKER_URL: "rediss://:@fully-qualified-hostname:6379/0?ssl_cert_reqs=CERT_OPTIONAL"

[JonnyWaffles]

Hi friends, I encountered the same issue after deploying encrypted Redis in transit/rest via AWS ElastiCache. Whenever I selected the broker tab my flower service died. Reading the source code I do not see the airflow celery flower command ever passing Airflow’s AIRFLOW__CELERY__SSL_ACTIVE (which eventually becomes broker_use_ssl) to the Flower command. Is this intentional? It’s rather confusing as it appears I need to configure flower independently of the airflow celery app. Thoughts?