Tasks in DAGs with `depends_on_past` or `task_concurrency` are not being scheduled
See original GitHub issueApache Airflow version: 1.10.13
What happened:
After performing an upgrade to v1.10.13 we noticed that tasks in some of our DAGs were not be scheduled. After a bit of investigation we discovered that by commenting out 'depends_on_past': True the issue went away.
What you expected to happen:
We think the issue might have something to do with this which was introduced to 1.10.13
[AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751)
How to reproduce it:
- Install Airflow v1.10.13 from pip
- Start webserver and scheduler
- Add the following code as a DAG
- Switch the DAG on in the UI.
from airflow import models
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'start_date': datetime(2018, 10, 31),
'depends_on_past': True,
'retries': 3,
'retry_delay': timedelta(minutes=5)
}
dag_name = 'my-test-dag'
with models.DAG(dag_name,
default_args=default_args,
schedule_interval='0 0 * * *',
catchup=False,
max_active_runs=5,
) as dag:
test = DummyOperator(
task_id='test'
)
Issue Analytics
- State:
- Created 3 years ago
- Comments:10 (8 by maintainers)
Top Results From Across the Web
Why my Airflow tasks got stuck in “no_status” and how I fixed it
One of our Airflow DAGs were not scheduling tasks. The issue looked very strange because it wasn't happening all the time.
Read more >Release Notes — Airflow Documentation
New to this release of Airflow is the concept of Datasets to Airflow, and with it a new way of scheduling dags: data-aware...
Read more >DAG getting stuck in "running" state indefinitely #15978 - GitHub
I expect all my tasks to be run and my dag to be marked as "success" or "failed" if there is an issue....
Read more >Airflow parallelism - Stack Overflow
parallelism is the max number of task instances that can run concurrently on airflow. This means that across all running DAGs, no more...
Read more >Airflow Task Parallelism. How to control concurrency
We can increase the concurrency of the task by increasing the number of schedulers. This will increase the task concurrency set at the...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found

https://github.com/apache/airflow/pull/12663 should fix it @nathadfield @mthoretton
I can confirm the bug. I was able to reproduce it with task with
task_concurrencyordepends_on_pastwithLocalExecutorand the following DAG: