question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support DAGS folder being in different location on scheduler and worker.

See original GitHub issue

Description

Dags folder in celery workers can be kept at different location as kept in Airflow scheduler node. Airflow task run can use dag full file path based on airflow home configuration on their node given dag file relative path to dags folder.

Use case / motivation

Currently scheduler uses absolute path for dags file to schedule task in task run command (-sd arg). This command makes hard to keep dags folder at different locations in celery workers which may be running in different environment. So using Airflow with celery with cross environment setup is difficult. This feature will help to keep dags folder at any location at worker nodes and tasks command can be configured for dag file path using home configuration.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:25 (20 by maintainers)

github_iconTop GitHub Comments

3reactions
ashbcommented, May 14, 2021

No, we should fix this properly in Airflow.

1reaction
barylukcommented, Nov 18, 2021

@potiuk Ok. I will add a note there, and send PR.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Scheduler — Airflow Documentation - Apache Airflow
The Airflow scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete. Behind the scenes, the scheduler...
Read more >
Add and update DAGs | Cloud Composer
Updating DAGs that run on a frequent schedule · Open the Logs tab for your environment in Google Cloud console. · Go to...
Read more >
Airflow not loading dags in /usr/local/airflow/dags
I find that I have to restart the scheduler for the UI to pick up the new dags, When I make changes to...
Read more >
Adding or updating DAGs - AWS Documentation
Amazon MWAA automatically syncs new and changed objects from your Amazon S3 bucket to Amazon MWAA scheduler and worker containers' /usr/local/airflow/dags ...
Read more >
7 Common Errors to Check When Debugging Airflow DAGs
This is why Airflow 2.2 introduced timetables as the new default scheduling method. Essentially, timetable is a DAG-level parameter that you ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found