Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TriggerDagRunOperator triggering subdags out of order

See original GitHub issue

Apache Airflow version: 2.1.1

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

What happened: We have dags that get triggered by another dag using the TriggerDagRunOperator. These dags include subdags as well. What we are seeing is that it triggers the subdags as well(without checking the dependencies) which should not be the case.

What you expected to happen: I would expect the dag run to be created for the parent dag only. The subdags should be run only when the upstream task dependencies are met.

How to reproduce it:

Create a dag that includes subdags with upstream task dependencies. Trigger it from another dag and see dag runs created for both subdags and main dag.

Anything else we need to know: Its because of this piece of code I believe https://github.com/apache/airflow/blob/main/airflow/api/common/experimental/trigger_dag.py#L81

which adds the subdags as well to the dags to be triggered. Any reason why this was done?

Issue Analytics

State:
Created 2 years ago
Comments:12 (4 by maintainers)

Top GitHub Comments

1reaction

uranusjrcommented, Aug 11, 2021

AIP-39 has not been released yet, so unless you’re running Airflow from the main branch, it’s not possible any of it would cause this.

1reaction

ManiBharatarajucommented, Jul 31, 2021

@enriqueayala - We created a custom TriggerDagRunOperator along with the changes in trigger_dag.py. I just copied the trigger dag run operator code and trigger_dag.py code and made the one change that I have mentioned. That worked. But I am unsure of why that was done or if that is the real reason behind this issue. As a temporary fix, u can do that.

With subdags having too many issues, we are planning to use taskgroups as a long-term fix.

Top Results From Across the Web

Want to use TriggerDagRunOperator in Airflow to trigger many ...

the TriggerDagRunOperator triggers a DAG run for a specified dag_id . This needs a trigger_dag_id with type string and a python_callable ...

airflow.operators.trigger_dagrun

Operator link for TriggerDagRunOperator. It allows users to access DAG triggered by task using TriggerDagRunOperator. name = Triggered DAG ...

The TriggerDagRunOperator in Airflow! Create DAG ...

Subdags, the ExternalTaskSensor or the TriggerDagRunOperator. One of the most common question is, how can I trigger another DAG B from my ...

Implement DAG Dependencies Using TriggerDagRun Operator

TriggerDagRunOperator is an effective way to implement cross-DAG dependencies. The operator allows to trigger other DAGs in the same Airflow environment.

TriggerDagRunOperator does not trigger dag on subsequent ...

Problem: The TriggerDagRunOperator does not trigger the target dag on subsequent run even with the reset_dag_run option set to true.