Problem with task_run and secrets mask

Apache Airflow version: 2.1.1

Kubernetes version: 1.20.2

Environment: Azure Kubernetes

What happened: I am upgrading from Airflow 2.0.2 -> 2.1+ On launching a simple Hello World dag (BashOperator task echoing “Hello World”), I encounter the following error:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 212, in task_run
    settings.configure_orm(disable_connection_pool=True)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 226, in configure_orm
    mask_secret(engine.url.password)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 103, in mask_secret
    _secrets_masker().add_mask(secret, name)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 117, in _secrets_masker
    raise RuntimeError("No SecretsMasker found!")
RuntimeError: No SecretsMasker found!

What you expected to happen: I expected the task to run as is I have tested this with Airflow 2.0.2 and it was running fine. Looking at the stack trace as already shared above, I believe the problem is with Secrets Masker.

I have already tried using the config variables AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS= True

From the code I see here:

def task_run(args, dag=None):
    """Runs a single task instance"""
    # Load custom airflow config
    if args.cfg_path:
        with open(args.cfg_path) as conf_file:
            conf_dict = json.load(conf_file)

        if os.path.exists(args.cfg_path):
            os.remove(args.cfg_path)

        conf.read_dict(conf_dict, source=args.cfg_path)
        settings.configure_vars()

    settings.MASK_SECRETS_IN_LOGS = True
    settings.configure_orm(disable_connection_pool=True)
    ...

The settings.MASK_SECRETS_IN_LOGS = True is overriding the user directive of AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS which should not happen.

I have tried debugging this issue by ssh’ing to the webserver

>>> settings.MASK_SECRETS_IN_LOGS = False
>>> settings.configure_orm(disable_connection_pool=True)
>>> exit()

and you can see there are no errors.

To conclude:

• SecretsMasker should not be enforced since this is breaking change, and should respect the user directive of AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS
• Will this be fixed?- I can raise a PR to address the issue

[potiuk]

I strongly believe the problem is that you had custom logging configuration done not according to the way we recommend it.

If you look there - if you want to have advanced logging configuration and add your own configuration, you should start your custom configuration with deepcopying of the original airflow configuration, and only add your changes after that:

https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration

The error that you see probably comes from copying the old logging from airlfow verbatim rather than deepcopying it dynamically in your custom configuration.

I will close it preemptively, in case the explanation is correct (in which case you can also get what you want with simply converting to the recommended way of configuring advanced logging.

Please comment here if my diagnosis is wrong and the reason was different, so that I can re-open it then,

[sudeepgupta90]

Thanks for all the help and inputs @potiuk @dave-martinez

We were using custom logging - have incorporated your feedback and now using the recommended approach, and its working as expected