Clarify how schedule_interval works

Hi,

This is a dummy example that consists of 4 tasks, back to back, all attached to the same DAG events_redshift. I’ve set schedule_interval to 1 for now, as I am trying to see this executed, but that’s not a real life example. This is running the CeleryExecutor and Postgresql.

"""
Extracts events from S3 and loads them into Redshift.
"""

from airflow import DAG 
from airflow.operators import DummyOperator
from datetime import datetime
from datetime import timedelta


default_args = { 
    'owner': 'airflow',
    'start_date': datetime(2015, 8, 5, 8, 4), 
    'schedule_interval': timedelta(minutes=1),
    'retry_delay': timedelta(minutes=1),
}

dag = DAG('events_redshift', default_args=default_args)

t_download_from_s3 = DummyOperator(
    task_id='download_from_s3',
    dag=dag,
)

t_cleanup = DummyOperator(
    task_id='cleanup',
    dag=dag,
)

t_upload_to_s3 = DummyOperator(
    task_id='upload_to_s3',
    dag=dag,
)

t_load_to_redshift = DummyOperator(
    task_id='load_to_redshift',
    dag=dag,
)

t_cleanup.set_upstream(t_download_from_s3)
t_upload_to_s3.set_upstream(t_cleanup)
t_load_to_redshift.set_upstream(t_upload_to_s3)

I can see the DAG on the web UI, however the only way to get it to execute the tasks is by clicking on it and Run manually, as you can see with download_from_s3.

This is the celery worker:

And the scheduler’s output, refreshing every 5 seconds.

My expectations are that this should be running every minute, and each task should be executed back to back, however none of this is happening.

So I guess my question is: do I have the wrong expectation, and what am I doing wrong?

Thanks a lot for your help!

[mistercrunch]

Oh. I should have caught this earlier but the issue is your DAG is actually a daily dag since at the moment schedule_interval is based on the argument you pass to the DAG object as in

dag = DAG("my_dag_id", schedule_interval=timedelta(hours=1))

I need to clarify that in the docs / API.

[mistercrunch]

Updated the tutorials / docs here: https://github.com/airbnb/airflow/pull/238