Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Improved SQL rendering within BigQueryInsertJobOperator
See original GitHub issueDescription
BigQueryInsertJobOperator requires the submission of a configuration parameter in the form of a dict. Unfortunately, if this contains a large SQL query - especially one that is formatted with new lines - then this cannot currently be rendered very nicely in the UI.
Use case / motivation
The issue with this is that it’s impossible to copy and paste the rendered query out of the Airflow UI, into a BigQuery browser and run it without lots of manual edits which is time wasted when troubleshooting problems.
Are you willing to submit a PR?
Yes. My current thought process around this would be to add an optional SQL parameter to the operator which, if provided, would be added into the configuration and could therefore have its own template field and SQL renderer.
e.g.
Issue Analytics
- State:
- Created 2 years ago
- Comments:22 (22 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Thanks @potiuk! Have been looking at this at times today and I’m definitely making some progress although it depends on whether @mehmax beats me to it!
A UI improvement seems like the right choice. https://github.com/apache/airflow/issues/17032#issuecomment-880795289
There are a ton of properties on job resource objects, so I’d be worried about the precedent for adding kwargs for each of them. Especially since query jobs aren’t the only kind of job