QuboleHook unable to build command with list tags
See original GitHub issueApache Airflow version: 1.10.10
Kubernetes version (if you are using kubernetes) (use kubectl version
):
Environment:
- Cloud provider or hardware configuration: Google Cloud
- OS (e.g. from /etc/os-release): Ubuntu 18.04.4 LTS
- Kernel (e.g.
uname -a
): Linux int-vidio-stats-airflow-bionic 5.0.0-1034-gcp #35-Ubuntu SMP Tue Mar 17 03:56:45 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux - Install tools: pip
- Others:
What happened:
Recently, we upgraded airflow version from v1.10.5 to v1.10.10 due to airflow instance that upgraded to ubuntu bionic. Existing dag configuration having tags as list causes an issue at some jobs that use Qubole. Error log:
Traceback (most recent call last):
File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/operators/qubole_operator.py", line 207, in execute
return self.get_hook().execute(context)
File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/hooks/qubole_hook.py", line 131, in execute
args = self.cls.parse(self.create_cmd_args(context))
File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/hooks/qubole_hook.py", line 248, in create_cmd_args
self._add_tags(tags, value)
File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/hooks/qubole_hook.py", line 267, in _add_tags
tags.extend(value)
AttributeError: 'set' object has no attribute 'extend'
What you expected to happen:
the tags should be included as a comma separated arguments to the build command.
How to reproduce it: using sample dag having list tags
t2 = QuboleOperator(
task_id='hive_gcs_location',
command_type="hivecmd",
script_location="s3n://public-qubole/qbol-library/scripts/show_table.hql",
notify=True,
tags=['tag1', 'tag2'],
trigger_rule="all_done"
)
and do an airflow run.
Issue Analytics
- State:
- Created 3 years ago
- Comments:7 (6 by maintainers)
Top Results From Across the Web
Source code for airflow.contrib.hooks.qubole_hook
Source code for airflow.contrib.hooks.qubole_hook ; log = logging ; COMMAND_CLASSES = { ; POSITIONAL_ARGS = { ; def flatten_list( ; def filter_options( ...
Read more >apache-airflow-providers-qubole 1.0.0rc1 - PyPI
Commit Committed Subject
b40dffa08 2020‑12‑08 Rename remaing modules to match AIP‑21 (#12917)
7825e8f59 2020‑11‑13 Docs installation improvements (#12304)
b2a28d159 2020‑11‑09 Moves provider packages scripts to dev...
Read more >Airflow: .../commits.rst | Fossies
This is detailed commit list of changes for versions provider package: qubole . ... [AIRFLOW-9347] Fix QuboleHook unable to add list to tags...
Read more >Changelog - Apache Airflow Documentation
Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859) ... Add documentation for CLI command airflow dags test (#8251).
Read more >airflow qubole 源码 - seaxiang
"""Qubole hook""" from __future__ import annotations import datetime import logging ... list], list]: """Build Command argument from command and options""" ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@chattarajoy I assigned you to these tickets.
sure @mik-laj, created issues for both the points mentioned will look into it. Could you pls change the assignee for the to me?