Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[dagster_airflow] run (containerized) dagster pipeline as a airflow task

See original GitHub issue

Use Case

Hi, currently dagster_airflow seems to only support converting dagster pipelines into tasks, but currently dagster pipelines can only be compiled to airflow dags.

Due to the airflow problems with SubDAGs (deadlocks), it would be much easier to integrate a dagster pipeline into airflow dags if it was possible to compile it to a task. This would also facilitate progressive migration where we start using dagster for part of the pipeline, and then we progressively add more task to it and remove them from the original airflow dag.

Ideas of Implementation

Similarly to the current make_airflow_dag and make_airflow_dag_containerized it would be nice to have a make_airflow_task and make_airflow_task_containerized which in turn create a PythonOperator and a KubernetesPodOperator.

Additional Info

While we would lose the granularity and control from Airflow by using this function, this would make it easier to integrate dagster pipelines with hybrid airflow dags.

Issue Analytics

State:
Created 3 years ago
Comments:5 (5 by maintainers)

Top GitHub Comments

1reaction

catherinewucommented, Nov 20, 2020

Hey @amarrella, cool I’ll mark this issue as closed for now. Feel free to re-open if you run into any issues!

Here is our 11/08/2020 community update on sensors: https://youtu.be/6nKCd2TaXik?t=2432. I don’t think we have plans to create a library of sensor integrations, but will definitely note the interest in google pubsub.

Thank you!

0reactions

sryzacommented, Aug 24, 2021

Looks like this was meant to be closed, but didn’t actually end up closed. Closing now.

Top Results From Across the Web

Execute on Airflow - Dagster Docs

A Dagster pipeline is first compiled with a set of config options into an execution plan, and then the individual execution steps are...

11 Running Tasks in Containers - Data Pipelines with Apache ...

Running containerized tasks in Airflow on Docker using the DockerOperator and on Kubernetes clusters using the KubernetesPodOperator.

Changeset View

**Containerized**: Tasks are executed in Docker containers. ... To compile this existing pipeline to Airflow, we'll use the `dagster-airflow` CLI tool.

Dabbling with Dagster vs. Airflow - Hacker News

Prefect is generally very easy to use. Essentially, you: (a) write a Python-based flow, which defines some job to run (with subtasks), (b)...

Dagster - Morioh

Developing data pipelines with Dagster makes testing easier and deploying ... When inspecting a run of your job, you can query over logs,...