question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Postgres Operator task doesn't sequence properly with TaskFlow API

See original GitHub issue

Apache Airflow version: 2.1.0.dev0

Kubernetes version (if you are using kubernetes) (use kubectl version): NA

Environment: Docker Apache/Airflow master-python3.6

What happened: some_pg_task it out of sequence

What you expected to happen:
Task sequence should be some_py_task >> some_pg_task

How to reproduce it:

from airflow.models import DAG
from airflow.providers.postgres.operators.postgres import PostgresOperator
import os

from airflow.utils.dates import days_ago

default_args = {'start_date': days_ago(1)}

dag_name = os.path.splitext(os.path.basename(__file__))[0]

with DAG(dag_name, default_args=default_args) as dag:

    @dag.task
    def some_py_task() -> str:
        return some_str

    some_str = some_py_task()

    some_pg_task = PostgresOperator(task_id='some_pg_task', 
                     sql='select * from %s',
                     parameters=[some_str],
                     postgres_conn_id='postgres_bob')

Anything else we need to know: The MSSQL Op behaves the same way

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
ephraimbuddycommented, Oct 11, 2021

This is what you need: test_pg_operator >> order_data >> order_summary >> load_task_flow There’s a difference between passing xcoms and creating dependency. We are working on documentation to clarify that. See : https://github.com/apache/airflow/issues/17686#issuecomment-937599461

Closing this for now

0reactions
ephraimbuddycommented, Oct 12, 2021
Read more comments on GitHub >

github_iconTop Results From Across the Web

TaskFlow API in Apache Airflow 2.0 — Should You Use It?
TaskFlow API is a feature that promises data sharing functionality and a simple interface for building data pipelines in Apache Airflow 2.0.
Read more >
Airflow: Running PostgresOperator within a PythonOperator ...
I am using the Taskflow API from Airflow 2.0. ... Somehow within a PythonOperator the PostgresOperator cannot make use of neither the param ......
Read more >
Working with TaskFlow — Airflow Documentation
This tutorial builds on the regular Airflow Tutorial and focuses specifically on writing data pipelines using the TaskFlow API paradigm which is introduced ......
Read more >
Airflow Postgres Operator 101: How to Connect and Execute ...
Airflow is a Task Automation tool. It helps organizations to schedule their tasks so that they are executed when the right time comes....
Read more >
TaskFlow API in Airflow 2.0 - YouTube
The TaskFlow API makes DAGs easier to write by abstracting the task de ... the new TaskFlow API features, as well as dive...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found