Webserver doesn't mask rendered fields for pending tasks

Apache Airflow version

2.2.5 (latest released)

What happened

When triggering a new dagrun the webserver will not mask secrets in the rendered fields for that dagrun’s tasks which didn’t start yet.

Tasks which have completed or are in state running are not affected by this.

What you think should happen instead

The webserver should mask all secrets for tasks which have started or not started.

.

How to reproduce

Create a variable my_secret and run this DAG

from datetime import timedelta

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.sensors.time_delta import TimeDeltaSensor
from airflow.utils.dates import days_ago

with DAG(
    "secrets",
    start_date=days_ago(1),
    schedule_interval=None,
) as dag:
    wait = TimeDeltaSensor(
        task_id="wait",
        delta=timedelta(minutes=1),
    )

    task = wait >> BashOperator(
        task_id="secret_task",
        bash_command="echo '{{ var.value.my_secret }}'",
    )

While the first task wait is running, displaying rendered fields for the second task secret_task will show the unmasked secret variable.

Operating System

Debian (Astronomer Airflow Docker image)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==1!3.2.0
apache-airflow-providers-cncf-kubernetes==1!3.0.0
apache-airflow-providers-elasticsearch==1!3.0.2
apache-airflow-providers-ftp==1!2.1.2
apache-airflow-providers-google==1!6.7.0
apache-airflow-providers-http==1!2.1.2
apache-airflow-providers-imap==1!2.2.3
apache-airflow-providers-microsoft-azure==1!3.7.2
apache-airflow-providers-mysql==1!2.2.3
apache-airflow-providers-postgres==1!4.1.0
apache-airflow-providers-redis==1!2.0.4
apache-airflow-providers-slack==1!4.2.3
apache-airflow-providers-sqlite==1!2.1.3
apache-airflow-providers-ssh==1!2.4.3

Deployment

Astronomer

Deployment details

No response

Anything else

We have seen this issue also in Airflow 2.2.3.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[tirkarthi]

Thanks, MASK_SECRETS_IN_LOGS should be set for masking of secret to work. It is set automatically when airflow tasks run is called. The sequential scheduler I use for local testing internally calls the airflow tasks run command to run tasks and thus I was confused over how masking is done when task is executed though the taskinstance.py file had not set the variable. I left a note in the PR over whether we should enable this just during rendering the task template in UI.

https://github.com/apache/airflow/pull/22754#issuecomment-1088945895

[hugochinchilla]

Actually this is a simplification, the problem I’m having is actually using data inside a Connection, but that connection is serialized as base64 in a factory that creates the operators, the code calls mask_secret() but the value is exposed until the task actually runs.

I’ve mitigated it creating a macro to call mask_secret() and return the value, so I can use something like {{ macros.plugin_name.mask_secret(some_secret) }} and the value is masked even when the task is pending.