question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Dynamic Tasks inside of TaskGroup do not have group_id prepended to task_id

See original GitHub issue

Apache Airflow version

2.3.3 (latest released)

What happened

As the title states, if you have dynamically mapped tasks inside of a TaskGroup, those tasks do not get the group_id prepended to their respective task_ids. This causes at least a couple of undesirable side effects:

  1. Task names are truncated in Grid/Graph* View. The tasks below are named plus_one and plus_two:

Screenshot from 2022-07-19 13-29-05 Screenshot from 2022-07-19 13-47-47

Presumably this is because the UI normally strips off the group_id prefix.

* Graph View was very inconsistent in my experience. Sometimes the names are truncated, and sometimes they render correctly. I haven’t figured out the pattern behind this behavior.

  1. Duplicate task_ids between groups result in a airflow.exceptions.DuplicateTaskIdFound, even if the group_id would normally disambiguate them.

What you think should happen instead

These dynamic tasks inside of a group should have the group_id prepended for consistent behavior.

How to reproduce

#!/usr/bin/env python3
import datetime

from airflow.decorators import dag, task
from airflow.utils.task_group import TaskGroup


@dag(
    start_date=datetime.datetime(2022, 7, 19),
    schedule_interval=None,
)
def test_dag():
    with TaskGroup(group_id='group'):
        @task
        def plus_one(x: int):
            return x + 1

        plus_one.expand(x=[1, 2, 3])

    with TaskGroup(group_id='ggg'):
        @task
        def plus_two(x: int):
            return x + 2

        plus_two.expand(x=[1, 2, 3])


dag = test_dag()


if __name__ == '__main__':
    dag.cli()

Operating System

CentOS Stream 8

Versions of Apache Airflow Providers

N/A

Deployment

Other

Deployment details

Standalone

Anything else

Possibly related: #12309

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:11 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
erdos2ncommented, Jul 21, 2022

Can I take this @bbovenzi ?

1reaction
bbovenzicommented, Jul 21, 2022

Ok I found the issue. If the groups are expanded on the initial page load the labels are correct. If a user clicks to expand a group the labels are not being updated. I’m creating a PR to fix this now.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Airflow task groups | Astronomer Documentation
Follow Astronomer's step-by-step guide to use task groups for organizing tasks within the graph view of the Airflow user interface.
Read more >
Airflow TaskGroups: All you need to know! - Marc Lamberti
Airflow TaskGroups provide a way to group your tasks and make your DAGs cleaner. Forget about SubDAGs and is discover TaskGroups!
Read more >
Airflow: Problem with creating dynamic task within TaskGroup
So I fixed this by creating TaskGroup dynamically within TaskGroup. Here's the code with TaskGroup('Review') as Review: data = [] filenames ...
Read more >
Dynamic Task Mapping — Airflow Documentation
Dynamic Task Mapping allows a way for a workflow to create a number of tasks at runtime based upon current data, rather than...
Read more >
Manage Dependencies Between Airflow Deployments, DAGs ...
Manage Dependencies Between Airflow Deployments, DAGs, and Tasks. 1.2K views Streamed 1 year ago. Astronomer. Astronomer. 3.65K subscribers.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found