Dynamic Tasks inside of TaskGroup do not have group_id prepended to task_id
See original GitHub issueApache Airflow version
2.3.3 (latest released)
What happened
As the title states, if you have dynamically mapped tasks inside of a TaskGroup
, those tasks do not get the group_id
prepended to their respective task_id
s. This causes at least a couple of undesirable side effects:
- Task names are truncated in Grid/Graph* View. The tasks below are named
plus_one
andplus_two
:
Presumably this is because the UI normally strips off the group_id
prefix.
* Graph View was very inconsistent in my experience. Sometimes the names are truncated, and sometimes they render correctly. I haven’t figured out the pattern behind this behavior.
- Duplicate
task_id
s between groups result in aairflow.exceptions.DuplicateTaskIdFound
, even if thegroup_id
would normally disambiguate them.
What you think should happen instead
These dynamic tasks inside of a group should have the group_id
prepended for consistent behavior.
How to reproduce
#!/usr/bin/env python3
import datetime
from airflow.decorators import dag, task
from airflow.utils.task_group import TaskGroup
@dag(
start_date=datetime.datetime(2022, 7, 19),
schedule_interval=None,
)
def test_dag():
with TaskGroup(group_id='group'):
@task
def plus_one(x: int):
return x + 1
plus_one.expand(x=[1, 2, 3])
with TaskGroup(group_id='ggg'):
@task
def plus_two(x: int):
return x + 2
plus_two.expand(x=[1, 2, 3])
dag = test_dag()
if __name__ == '__main__':
dag.cli()
Operating System
CentOS Stream 8
Versions of Apache Airflow Providers
N/A
Deployment
Other
Deployment details
Standalone
Anything else
Possibly related: #12309
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
Issue Analytics
- State:
- Created a year ago
- Comments:11 (9 by maintainers)
Top Results From Across the Web
Airflow task groups | Astronomer Documentation
Follow Astronomer's step-by-step guide to use task groups for organizing tasks within the graph view of the Airflow user interface.
Read more >Airflow TaskGroups: All you need to know! - Marc Lamberti
Airflow TaskGroups provide a way to group your tasks and make your DAGs cleaner. Forget about SubDAGs and is discover TaskGroups!
Read more >Airflow: Problem with creating dynamic task within TaskGroup
So I fixed this by creating TaskGroup dynamically within TaskGroup. Here's the code with TaskGroup('Review') as Review: data = [] filenames ...
Read more >Dynamic Task Mapping — Airflow Documentation
Dynamic Task Mapping allows a way for a workflow to create a number of tasks at runtime based upon current data, rather than...
Read more >Manage Dependencies Between Airflow Deployments, DAGs ...
Manage Dependencies Between Airflow Deployments, DAGs, and Tasks. 1.2K views Streamed 1 year ago. Astronomer. Astronomer. 3.65K subscribers.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Can I take this @bbovenzi ?
Ok I found the issue. If the groups are expanded on the initial page load the labels are correct. If a user clicks to expand a group the labels are not being updated. I’m creating a PR to fix this now.