Handling broken pipelines in dagit
See original GitHub issueRight now when something is wrong w/ a pipeline definition, dagit
will spew the error to the console and refuse to load.
Instead, it would be great to show the dagit UI and highlight which pipeline in the repository is broken
Issue Analytics
- State:
- Created 4 years ago
- Comments:10 (7 by maintainers)
Top Results From Across the Web
A Dagster Crash Course
They're usually tasked with “cleaning data”, fixing broken pipelines in the middle of the night, and are often on-call for questions and ...
Read more >How to avoid running the rest of a dagster pipeline under ...
To achieve this result, I raise an error when the data meets the invalid condition so the pipeline stops and the rest of...
Read more >D5034 [dagit] Gracefully handle repo location failures on start
Load workspaces that have no repos, broken repos, or only working repos. Verify that the error appears on Dagit startup for broken repos,...
Read more >Debugging Broken Data Pipelines with Airflow - YouTube
Session presented by Francisco Alberini at Airflow Summit 2022“Why is my data missing?” “Why didn't my Airflow job run?
Read more >Deploying Dagster to AWS - Medium
Dagit can also be used to launch pipeline runs and check their results, as well as all ... an invalid input can break...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I ran across an issue that seems related. Going over the Inputs tutorial at https://dagster.readthedocs.io/en/stable/sections/learn/tutorial/inputs.html I ran the test pipeline first with an untyped solid, and then changed the same solid to use type signatures. The
runs
view refused to load with a GraphQL error:GraphQL error] Pipeline "hello_inputs_pipeline" config errors: Error 1: Type failure at path "root:solids:add_hello_to_word:inputs:word:value" on type "String". Value at path root:solids:add_hello_to_word:inputs:word:value is not valid. Expected "String".
The backtrace I see in the console is:
This is somewhat annoying, since making type changes to your solids that are incompatible with previous runs in your history will blow up Dagit, until you manually excise the problematic runs from history (haven’t tried this yet, so not sure if it’s possible). Is this the same problem this issue is addressing and should create a new one?
Closing in favor of #2718.