Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Code reuse with Dags

See original GitHub issue

I’d like clone a dag multiple times to replicate the logic for multiple similar markets/sources.

I drive most of my dag configuration by pulling json Variables to parametrize values, so having configuration for each market isnt a problem. My problem is actually cloning the dag+tasks without actually having two seperate scripts containing the same code (albeit with different dag names).

Is this possible or does it go against the design pattern of aiflow dags?

Issue Analytics

State:
Created 8 years ago
Comments:8 (5 by maintainers)

Top GitHub Comments

2reactions

jlowincommented, Sep 29, 2015

For folks generating multiple dags in a script, maybe a DagContainer object (which is just a list subclass) would be nice… then in addition to any DAG objects found in a script, anything in any DagContainers is also imported. Not sure how many people are actually doing this but I know it’s come up a few times.

1reaction

mistercrunchcommented, Sep 29, 2015

It’s actually probably cleaner to write a function or a class that generates dag objects base on input parameters

Top Results From Across the Web

Assign airflow task to several DAGs - Stack Overflow

I wonder if there is any other approach to reuse an existing airflow task assign to a different dag. I am using apache-airflow[docker,kubernetes]== ......

Re: reuse dags - The Mail Archive

I am looking for a built-in simple way to > import dags. > > Reusing and factorizing code is quite common need.

Dynamically generating DAGs in Airflow - Astronomer Docs

Get to know the best ways to dynamically generate DAGs in Apache Airflow. Use examples to generate DAGs using single- and multiple-file methods....

Best Practices - Apache Airflow

Creating a new DAG is a three-step process: writing Python code to create a DAG object,. testing if the code meets our expectations,....

Code reuse - Wikipedia

In software development code reuse, also called software reuse, is the use of existing software, or software knowledge, to build new software, :...