Helm Chart Allow for Empty data settings

The helm-chart should not force the settings of Resultsbackend, Metadata and Broker. Either through an additional configuration option or implicitly by not providing values, the helm-chart should refrain from populating those Airflow-Env configurations.

Use case / motivation The current Helm chart forces the configuration of certain values such as brokerUrl (https://github.com/apache/airflow/blob/main/chart/values.yaml#L236) Which will then be populated as Airflow ENV config. Due to Airflows configuration precedence (https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html) are these settings taking highest priority in Airflow configuration.

With the suggested change, users can dynamically provide Airflow configurations through Airflow’s __CMD environment variables.

In other words; provide users with the option to utilize Airflow’s __CMD ENV configuration.

I am willing to submit a PR.

[dstandish]

I too see the value in allowing components of the standard_airflow_environment to be sourced in a way other than k8s secret.

One reason is, if it’s an officially supported way of managing configuration in airflow, then the helm chart should probably allow it. Otherwise it’s confusing, as a user sees “oh i can store these params in AWS ssm” only to later discover it’s not possible with the helm chart.

Another reason is the convenience of managing everything from one source. Rather than some things in k8s secrets, others in vault or SSM – better to have one place.

And from a CI perspective, managing k8s secrets can be a little more cumbersome than something like vault or SSM.

From an implementation perspective, it seems we might be able to just move everything to extraEnvFrom (or something like this), and get rid of the standard_airflow_environment e.g. and the similar celery one. Then the user has the flexibility afforded by airflow.

[potiuk]

Quite agree. I think Airflow precedence makes things easier at times but using pre-defined env variables in chart makes it less powerful and more opinionated.

I prepared a PR that might be a good way of addressing the issue: #18974

WDYT?