Passing a jinja template in TriggerDagRunOperator conf raises json.decoder.JSONDecodeError

Apache Airflow version:2.0.1

Environment:

• OS (e.g. from /etc/os-release): Ubuntu

What happened: When having a PythonOperator that returns xcom parameters to a TriggerDagRunOperator like in this non-working example:

def conditionally_trigger(spider):
    is_ok = db.query()
    if is_ok:
        payload = {
            "foo": "foo",
            "bar": "bar",
        }
        return payload
    else:
        raise AirflowSkipException


is_job_ok_to_trigger = PythonOperator(
    task_id='task1',
    python_callable=conditionally_trigger,
    dag=dag,
    do_xcom_push=True
)

trigger_another_dag= TriggerDagRunOperator(
    task_id='trigger_another_dag',
    trigger_dag_id='another_dag',
    dag=dag,
    conf="{{ ti.xcom_pull(task_ids='task1', key='return_value') }}",  
)

is_job_ok_to_trigger >> trigger_another_dag

With the first task returning this xcom:

In the second task we fail with this error message:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/trigger_dagrun.py", line 135, in execute
    replace_microseconds=False,
  File "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 123, in trigger_dag
    replace_microseconds=replace_microseconds,
  File "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 78, in _trigger_dag
    run_conf = conf if isinstance(conf, dict) else json.loads(conf)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

with a rendered template of conf being:

What you expected to happen: We should be able to forward a dict like string from an upstream task to TriggerDagRunOperator conf template field which is then converted to json.

It might be related to https://github.com/apache/airflow/issues/14619

How to reproduce it:

from airflow import DAG

from airflow.operators.python_operator import PythonOperator
from airflow.operators.trigger_dagrun import TriggerDagRunOperator

dag = DAG('foo', schedule_interval='@daily')

def conditionally_trigger():
    payload = {
        "foo": "foo",
        "bar": "bar",
    }
    return payload

is_job_ok_to_trigger = PythonOperator(
    task_id='task1',
    python_callable=conditionally_trigger,
    dag=dag,
    do_xcom_push=True
)

trigger_another_dag = TriggerDagRunOperator(
    task_id='trigger_another_dag',
    trigger_dag_id='another_dag',
    dag=dag,
    conf="{{ ti.xcom_pull(task_ids='task1', key='return_value') }}",  
)

is_job_ok_to_trigger >> trigger_another_dag

Anything else we need to know:

[eladkal]

Should be solved by https://github.com/apache/airflow/pull/14603

[mik-laj]

Here is workaround: https://github.com/apache/airflow/issues/11983