Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Airflow scheduler hangs when loading pipeline

See original GitHub issue

Wrote my first dagster pipeline yesterday and wanted to run it on Airflow. However it was not showing up on Airflow as the scheduler process in charge of processing the DAG seemed to disappear.

After some manual debugging, found the cause was PDB instance on the dagster.utils module that was being imported when trying to load the Airflow DAG.

Seems the code was added here but I haven’t been able to find any reference to the PDB so I’m wondering if that instance could be hidden behind a conditional flag to avoid Scheduler hangs on Airflow.

Great product btw!

Issue Analytics

State:
Created 4 years ago
Comments:6 (5 by maintainers)

Top GitHub Comments

1reaction

casassgcommented, Feb 4, 2020

Appreciate the fast fix. I actually commented it out on my machine and was able to run the DAG successfully. Thanks for the support 👍

1reaction

helloworldcommented, Feb 4, 2020

@casassg just landed a fix in master so that ForkedPdb won’t be imported automatically.

Our next release will be on Tuesday, so if you need the fix now I’d recommend simply deleting the import in dagster/python_modules/dagster/dagster/utils/__init__.py or doing an editable install of the master branch.

Top Results From Across the Web

scheduler gets stuck without a trace #7935 - apache/airflow

I'm running Airflow 1.10.4 as Celery in k8s. The scheduler pod is getting stuck while starting up at the step 'Resetting orphaned tasks'....

Troubleshooting Airflow scheduler issues | Cloud Composer

Low performance of the Airflow database might be the reason why the scheduler is slow.

Airflow scheduler is slow to schedule subsequent tasks

To get the time Airflow is spending loading Dags, run the command: ... a single task so you do not suffer from the...

7 Common Errors to Check When Debugging Airflow DAGs

1. Your DAG Isn't Running at the Expected Time · Airflow's Schedule Interval · Use Timetables for Simpler Scheduling · Airflow Time Zones....

Scheduler — Airflow Documentation - Apache Airflow

The Airflow scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete. Behind the scenes, the scheduler...