Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

DAGs with Dynamic Task Mapping are not rendered when cluster policies overwrite a parameter

See original GitHub issue

Apache Airflow version

2.3.2 (latest released)

What happened

Using cluster policies, I set a default on_failure_callback. This works great with all tasks expect with task new task mapping.

What you think should happen instead

Policies should work on all tasks, including the dynamic mapping or generate an exception with the error.

How to reproduce

Overwrite any parameters using the task_policy.

Operating System

Official docker image

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project’s Code of Conduct

Issue Analytics

State:
Created a year ago
Comments:6 (5 by maintainers)

Top GitHub Comments

1reaction

uranusjrcommented, Dec 8, 2022

So I’m relabelling this to a feature since everything is working “as we designed it” and nothing’s out of ordinary, but our current design does not meet user expectation and should be improved. This also means the fix would not need to constrain 2.5.1, although I’m still keeping it in the milestone since it’d be totally fine to backport the feature if it’s ready by then.

The simplest fix to this would be to implement setters for the on_*_callback callbacks on MappedOperator. A pull request would be much appreciated.

1reaction

uranusjrcommented, Dec 8, 2022

Ah OK, so the main issue is the callback property being read-only. This is not actually related to attr.define; on_failure_callback is a property, not a plain attribute, and we don’t define any setters for those properties.

Top Results From Across the Web

Dynamic Task Mapping — Airflow Documentation

Dynamic Task Mapping allows a way for a workflow to create a number of tasks at runtime based upon current data, rather than...

Airflow 2.5.1 Milestone - GitHub

DAGs with Dynamic Task Mapping are not rendered when cluster policies overwrite a parameter AIP-42 Dynamic Task Mapping area:core kind:feature Feature ...

Concepts - Apache Airflow documentation

Dynamic Task Mapping ... Assigning multiple parameters to a non-TaskFlow operator · Filtering items from an expanded ... Priority Weights · Cluster Policies....

Create dynamic Airflow tasks | Astronomer Documentation

With the release of Airflow 2.3, you can write DAGs that dynamically generate parallel tasks at runtime. This feature, known as dynamic task...

Task definition parameters - Amazon ECS - AWS Documentation

Required: no. The Docker networking mode to use for the containers in the task. For Amazon ECS tasks hosted on Fargate, the awsvpc...