successful DAG run fails to be scheduled after being manually cleared if Dag.dagrun_timeout is set

Apache Airflow version: 2.0.0

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): redhat 7.9
• Kernel (e.g. uname -a): 3.10.0-1160.11.1.el7.x86_64
• Install tools: pip
• Others:

What happened:

I cleared a success DAG run and it failed to be scheduled again with the following error message:

{scheduler_job.py:1639} INFO - Run scheduled__2021-02-02T16:00:00+00:00 of some_job has timed-out

After removing dagrun_timeout, the same dag run can be rescheduled.

What you expected to happen:

I expect “timeout” is counted from the moment the DAG is reset.

Anything else we need to know:

I can reproduce it with the following code. Not sure if it is the intended behavior.

from airflow import DAG
from airflow.operators.dummy import DummyOperator
import pendulum
with DAG("test_timeout",
    default_args={'owner': 'airflow'},
    start_date= pendulum.yesterday(),
    schedule_interval='@daily',
    # add the following parameter after the first run is complete and then clear the success run
    # dagrun_timeout=pendulum.duration(minutes=1),
) as dag:
    DummyOperator(task_id='dummy')

[mattellis]

@YangMuye @kaxil Our users have the same issue. In case it’s of use, I figured out that the models.dag.clear method being invoked when a dag run is cleared from the graph or tree view, does not set the activate_dag_runs flag to True in the models.taskinstance.clear_task_instances method, which is where the start_date value for the run is reset to now(): https://github.com/apache/airflow/blob/09327ba6b371aa68cf681747c73a7a0f4968c173/airflow/models/dag.py#L1328

if activate_dag_runs and tis:
...
    dr.start_date = timezone.utcnow()

https://github.com/apache/airflow/blob/09327ba6b371aa68cf681747c73a7a0f4968c173/airflow/models/taskinstance.py#L221

Alternatively, the browse DAG runs view uses a different path to clear dag runs, and is still effective in resetting the dagrun start date: https://github.com/apache/airflow/blob/09327ba6b371aa68cf681747c73a7a0f4968c173/airflow/www/views.py#L3429

We’ve advised our users to use this method as a workaround for now, and set our normal 24 hour dag_run timeout to 7 days to cover most common use cases of clearing recently failed runs.

[kaxil]

Closing – fixed by https://github.com/apache/airflow/pull/16401 – which will be part of Airflow 2.1.3