Extend elastic DAG with a binary tree, grid, star

Description

We have “elastic DAG” which we use in various performance tests. It allows you to generate any DAG using environment variables. An example test that uses such a DAG looks like this

        with mock.patch.dict("os.environ", {
            "PERF_DAGS_COUNT": str(dag_count),
            "PERF_TASKS_COUNT": str(task_count),
            "PERF_START_AGO": start_ago,
            "PERF_SCHEDULE_INTERVAL": schedule_interval,
            "PERF_SHAPE": shape,
        }), conf_vars({
            ('scheduler', 'use_job_schedule'): 'True',
        }):
            dagbag = DagBag(dag_folder=ELASTIC_DAG_FILE, include_examples=False)
            with assert_queries_count(expected_query_count):
                processor = DagFileProcessor([], mock.MagicMock())
                processor._process_dags(dagbag.dags.values())

Source: https://github.com/apache/airflow/blob/6bd2e59/tests/jobs/test_scheduler_job.py#L1175

This DAG File allows you to generate DAGs that contain all the tasks connected in a linear manner or have no connections.

It would be useful to add more complex DAGs, especially binary trees or/and grids or/and stars. This will allow us to test more complex cases more easily.

Use case / motivation

N/A

Related Issues

N/A

[mik-laj]

All children of task 1. This is needed to check if the tasks will be performed at the second iteration of the loop. Grid is needed to perform a stress test and at the same time not get lost in all dependencies. Binary tree will allow us to check cases more real to real situations.

[alexandrecaze]

Would we have a second layer for the star ?

or all children of task 1 ?