Support DAGS folder being in different location on scheduler and worker.
See original GitHub issueDescription
Dags folder in celery workers can be kept at different location as kept in Airflow scheduler node. Airflow task run can use dag full file path based on airflow home configuration on their node given dag file relative path to dags folder.
Use case / motivation
Currently scheduler uses absolute path for dags file to schedule task in task run command (-sd
arg). This command makes hard to keep dags folder at different locations in celery workers which may be running in different environment. So using Airflow with celery with cross environment setup is difficult.
This feature will help to keep dags folder at any location at worker nodes and tasks command can be configured for dag file path using home configuration.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:25 (20 by maintainers)
Top Results From Across the Web
Scheduler — Airflow Documentation - Apache Airflow
The Airflow scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete. Behind the scenes, the scheduler...
Read more >Add and update DAGs | Cloud Composer
Updating DAGs that run on a frequent schedule · Open the Logs tab for your environment in Google Cloud console. · Go to...
Read more >Airflow not loading dags in /usr/local/airflow/dags
I find that I have to restart the scheduler for the UI to pick up the new dags, When I make changes to...
Read more >Adding or updating DAGs - AWS Documentation
Amazon MWAA automatically syncs new and changed objects from your Amazon S3 bucket to Amazon MWAA scheduler and worker containers' /usr/local/airflow/dags ...
Read more >7 Common Errors to Check When Debugging Airflow DAGs
This is why Airflow 2.2 introduced timetables as the new default scheduling method. Essentially, timetable is a DAG-level parameter that you ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
No, we should fix this properly in Airflow.
@potiuk Ok. I will add a note there, and send PR.