Nested templated variables do not always render

Apache Airflow version: 1.10.14 and 1.10.8.

Environment: Python 3.6 and Airflow 1.10.14 on sqllite,

What happened:

Nested jinja templates do not consistently render when running tasks. TI run rendering behavior also differs from airflow UI and airflow render cli.

What you expected to happen:

Airflow should render nested jinja templates consistently and completely across each interface. Coming from airflow 1.8.2, this used to be the case.

This regression may have been introduced in 1.10.6 with a refactor of BaseOperator templating functionality. https://github.com/apache/airflow/pull/5461

Whether or not a nested layer renders seems to differ based on which arg is being templated in an operator and perhaps order. Furthermore, it seems like the render cli and airflow ui each apply TI.render_templates() a second time, creating inconsistency in what nested templates get rendered.

There may be bug in the way BaseOperator.render_template() observes/caches templated fields

How to reproduce it: From the most basic airflow setup

nested_template_bug.py

from datetime import datetime
from airflow import DAG
from airflow.operators.python_operator import PythonOperator

with DAG("nested_template_bug", start_date=datetime(2021, 1, 1)) as dag:

    arg0 = 'level_0_{{task.task_id}}_{{ds}}'
    kwarg1 = 'level_1_{{task.op_args[0]}}'

    def print_fields(arg0, kwarg1):
        print(f'level 0 arg0: {arg0}')
        print(f'level 1 kwarg1: {kwarg1}') 

    nested_render = PythonOperator(
        task_id='nested_render',
        python_callable=print_fields,
        op_args=[arg0, ],
        op_kwargs={
            'kwarg1': kwarg1,
        },
    )

> airflow test c

level 0 arg0: level_0_nested_render_2021-01-01
level 1 kwarg1: level_1_level_0_{{task.task_id}}_{{ds}}

> airflow render nested_template_bug nested_render 2021-01-01

# ----------------------------------------------------------
# property: op_args
# ----------------------------------------------------------
['level_0_nested_render_2021-01-01']

# ----------------------------------------------------------
# property: op_kwargs
# ----------------------------------------------------------
{'kwarg1': 'level_1_level_0_nested_render_2021-01-01'}

[uranusjr]

If we support nested rendering we’d also need to worry about circular reference, which would require a significantly more elaborated algorithm to properly handle. Dependency resolution is hell, don’t go there if at all possible.

[potiuk]

I looked at this and the problem turned out to be a bit different.

True - we should not even attempt to solve the recursive rendering of fields cross-referencing each other. But this was not the case. This was a kwargs field which referred opargs field - and kwargs was AFTER opargs in the list of templated fields, so it should work in principle, without resolving the recursive problem.

I looked at it and it turned out, this was a problem introduced by #8805 - where instead of the original task we started to use a copy of the task locked for execution. Unfortunately - the task in context was still the original one, not the copy, which caused the rendering problem (as rendered fields in the ‘context’ task were not updated at all.

This means that if someone would use the context['task'] later in Python Callable or custom operator, the fields were not rendered there either 😱 .

I fixed that in the #18516 by replacing the task in context with the same locked-down copy - not sure if this might have other side effects (but I think it could only be positive side effects 😄 ).