Postgres Operator task doesn't sequence properly with TaskFlow API
See original GitHub issueApache Airflow version: 2.1.0.dev0
Kubernetes version (if you are using kubernetes) (use kubectl version
):
NA
Environment: Docker Apache/Airflow master-python3.6
What happened: some_pg_task it out of sequence
What you expected to happen:
Task sequence should be
some_py_task >> some_pg_task
How to reproduce it:
from airflow.models import DAG
from airflow.providers.postgres.operators.postgres import PostgresOperator
import os
from airflow.utils.dates import days_ago
default_args = {'start_date': days_ago(1)}
dag_name = os.path.splitext(os.path.basename(__file__))[0]
with DAG(dag_name, default_args=default_args) as dag:
@dag.task
def some_py_task() -> str:
return some_str
some_str = some_py_task()
some_pg_task = PostgresOperator(task_id='some_pg_task',
sql='select * from %s',
parameters=[some_str],
postgres_conn_id='postgres_bob')
Anything else we need to know: The MSSQL Op behaves the same way
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:6 (4 by maintainers)
Top Results From Across the Web
TaskFlow API in Apache Airflow 2.0 — Should You Use It?
TaskFlow API is a feature that promises data sharing functionality and a simple interface for building data pipelines in Apache Airflow 2.0.
Read more >Airflow: Running PostgresOperator within a PythonOperator ...
I am using the Taskflow API from Airflow 2.0. ... Somehow within a PythonOperator the PostgresOperator cannot make use of neither the param ......
Read more >Working with TaskFlow — Airflow Documentation
This tutorial builds on the regular Airflow Tutorial and focuses specifically on writing data pipelines using the TaskFlow API paradigm which is introduced ......
Read more >Airflow Postgres Operator 101: How to Connect and Execute ...
Airflow is a Task Automation tool. It helps organizations to schedule their tasks so that they are executed when the right time comes....
Read more >TaskFlow API in Airflow 2.0 - YouTube
The TaskFlow API makes DAGs easier to write by abstracting the task de ... the new TaskFlow API features, as well as dive...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
This is what you need:
test_pg_operator >> order_data >> order_summary >> load_task_flow
There’s a difference between passing xcoms and creating dependency. We are working on documentation to clarify that. See : https://github.com/apache/airflow/issues/17686#issuecomment-937599461Closing this for now
cc: @alex-astronomer