Import error when using custom backend and sql_alchemy_conn_secret

Apache Airflow version: 2.0.0

Environment:

• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): custom Docker image (FROM python:3.6) and macOS Big Sur (11.0.1)
• Kernel (e.g. uname -a):
  • Linux xxx 4.14.174+ #1 SMP x86_64 GNU/Linux
  • Darwin xxx 20.1.0 Darwin Kernel Version 20.1.0 rRELEASE_X86_64 x86_64
• Install tools:
• Others:

What happened:

I may have mixed 2 different issues here, but this is what happened to me.

I’m trying to use Airflow with the airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend and a sql_alchemy_conn_secret too, however, I have a NameError exception when attempting to run either airflow scheduler or airflow webserver:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 786, in <module>
    conf.read(AIRFLOW_CONFIG)
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 447, in read
    self._validate()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 196, in _validate
    self._validate_config_dependencies()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 224, in _validate_config_dependencies
    is_sqlite = "sqlite" in self.get('core', 'sql_alchemy_conn')
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 324, in get
    option = self._get_option_from_secrets(deprecated_key, deprecated_section, key, section)
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 342, in _get_option_from_secrets
    option = self._get_secret_option(section, key)
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 303, in _get_secret_option
    return _get_config_value_from_secret_backend(secrets_path)
NameError: name '_get_config_value_from_secret_backend' is not defined

What you expected to happen:

A proper import and configuration creation.

How to reproduce it:

airflow.cfg:

[core]

# ...

sql_alchemy_conn_secret = some-key

# ...

[secrets]

backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

backend_kwargs = { ... }

# ...

Anything else we need to know:

Here is the workaround I have for the moment, not sure it works all the way, and probably doesn’t cover all edge cases, tho it kinda works for my setup:

Move get_custom_secret_backend before (for me it’s actually below _get_config_value_from_secret_backend): https://github.com/apache/airflow/blob/cc87caa0ce0b31aa29df7bbe90bdcc2426d80ff1/airflow/configuration.py#L794

Then comment: https://github.com/apache/airflow/blob/cc87caa0ce0b31aa29df7bbe90bdcc2426d80ff1/airflow/configuration.py#L232-L236

[potiuk]

Thanks for verifying 😃.

I think we will be looking at releasing 2.0.1 in early January. Xmas/EOY time is a bit quiet, but we already have a few annoying ‘teething problems’ with 2.0 reported and some of them already fixed (some of them need discussion/fix) still. There are not many, but I think also there will be some users who will use the “quiet” period around Xmas Holidays to test migration to 2.0 and report some more issues they find. Most of big businesses have moratorium for production changes mid-Dec, beginning of Jan which results in more time to play with staging systems 😃.

You can see all the opened (and closed as well) issues for 2.0.1 here: https://github.com/apache/airflow/milestone/23

We will likely decide on January 6th at our next DevCall (see calendar here: https://calendar.google.com/calendar/embed?src=c_dhdh3bdjc42c4ngtnpg7ovcs9g%40group.calendar.google.com )

[potiuk]

I pushed a possible fix in #13260