question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Extending SubDAGOperator doesn't show Zoom into SubDAG in the UI

See original GitHub issue

Apache Airflow version

2.1.x

Operating System

Debian GNU/Linux 10 (buster)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Breeze

What happened

I extended the SubDAGOperator to add conf to template_fields as a workaround for #18491.

Extending the Zoom into SubDAG option in the UI doesn’t appear.

What you expected to happen

Zoom into SubDAG option should be available for Extended SubDAGOperators

How to reproduce

  1. Use the following DAG
Click to expand!
# [START example_subdag_operator]
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.subdag import SubDagOperator
from airflow.utils.dates import days_ago
from datetime import datetime

DAG_NAME = 'example_subdag_operator'

args = {
    'owner': 'airflow',
}


class CustomSubDagOperator(SubDagOperator):
    template_fields = SubDagOperator.template_fields + ("conf",)

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._task_type = 'SubDagOperator'

    @property
    def task_type(self):
        return self._task_type

def subddag_python(dag_run, **kwargs):
    print (kwargs)
    print (f"{dag_run.dag_id} {dag_run.conf}")

def subdag(parent_dag_name, child_dag_name, args):
    """
    Generate a DAG to be used as a subdag.

    :param str parent_dag_name: Id of the parent DAG
    :param str child_dag_name: Id of the child DAG
    :param dict args: Default arguments to provide to the subdag
    :return: DAG to use as a subdag
    :rtype: airflow.models.DAG
    """
    dag_subdag = DAG(
        dag_id=f'{parent_dag_name}.{child_dag_name}',
        default_args=args,
        start_date=datetime(2021, 1, 1),
        catchup=False,
        schedule_interval="@daily",
    )

    tn = PythonOperator(
            task_id=f'subdag_python',
            python_callable=subddag_python,  # make sure you don't include the () of the function
            provide_context=True,
            dag=dag_subdag
    )

    return dag_subdag



def simple_python(dag_run):
    print (f"{dag_run.dag_id} {dag_run.conf}")

with DAG(
    dag_id=DAG_NAME, default_args=args, start_date=days_ago(2), schedule_interval="@once", tags=['example']
) as dag:

    tn = PythonOperator(
            task_id=f'simple_python',
            python_callable=simple_python,  # make sure you don't include the () of the function
            provide_context=True,
    )

    section_1 = CustomSubDagOperator(
        task_id='section-1',
        subdag=subdag(DAG_NAME, 'section-1', args),
        # conf="{{ dag_run.conf }}"
    )

    section_2 = SubDagOperator(
        task_id='section-2',
        subdag=subdag(DAG_NAME, 'section-2', args),
        # conf="{{ dag_run.conf }}"
    )

    tn >> section_1
    tn >> section_2

  1. On the airflow UI you will see Zoom into SubDAG option for the direct task but not for the extended task

Anything else

In the tree view call modal, we are directly checking for the operator, which is probably the issue.

https://github.com/apache/airflow/blob/56a69b7c3e4fb3984b1e509ce2880c427a86e6ba/airflow/www/static/js/tree.js#L308

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
uranusjrcommented, Oct 5, 2021

We can “fake” identity by serialising MRO information and implement __instancecheck__ and __subclasscheck__ on BaseOperator. But honestly my suggestion would be to avoid class introspection entirely. Instead of checking for the operator type, we should add an attribute (zoomable or something) and set it to True on SubDagOperator.

1reaction
bhavaniravicommented, Sep 28, 2021

Done @bbovenzi

Read more comments on GitHub >

github_iconTop Results From Across the Web

Airflow: how to extend SubDagOperator? - Stack Overflow
In fact, most things work without this trick as far as I can tell except the "Zoom into Subdag" button in the UI....
Read more >
[jira] [Updated] (AIRFLOW-1025) Zoom into SubDAG doesn't ...
When I click on the subdag operator and "Zoom > into Sub DAG" it doesn't show these DAG runs. (It used to prior...
Read more >
Airflow SubDAGs | Astronomer Documentation
SubDAGs are a legacy Airflow feature that allowed the creation of reusable task patterns in DAGs. SubDAGs caused performance and functional issues, and...
Read more >
Concepts — Airflow Documentation - Apache Airflow
For example, a common pattern with SubDagOperator is to define the subdag ... You can zoom into a SubDagOperator from the graph view...
Read more >
Apache Airflow Documentation - SpecialistOff.NET
The Airflow UI makes it easy to monitor and troubleshoot your data pipelines. ... You can zoom into a SubDagOperator from the graph...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found