question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

QuboleHook unable to build command with list tags

See original GitHub issue

Apache Airflow version: 1.10.10

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

  • Cloud provider or hardware configuration: Google Cloud
  • OS (e.g. from /etc/os-release): Ubuntu 18.04.4 LTS
  • Kernel (e.g. uname -a): Linux int-vidio-stats-airflow-bionic 5.0.0-1034-gcp #35-Ubuntu SMP Tue Mar 17 03:56:45 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: pip
  • Others:

What happened:

Recently, we upgraded airflow version from v1.10.5 to v1.10.10 due to airflow instance that upgraded to ubuntu bionic. Existing dag configuration having tags as list causes an issue at some jobs that use Qubole. Error log:

Traceback (most recent call last):
  File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/operators/qubole_operator.py", line 207, in execute
    return self.get_hook().execute(context)
  File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/hooks/qubole_hook.py", line 131, in execute
    args = self.cls.parse(self.create_cmd_args(context))
  File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/hooks/qubole_hook.py", line 248, in create_cmd_args
    self._add_tags(tags, value)
  File "/home/jenkins/virtualenv/airflow/lib/python3.6/site-packages/airflow/contrib/hooks/qubole_hook.py", line 267, in _add_tags
    tags.extend(value)
AttributeError: 'set' object has no attribute 'extend'

What you expected to happen:

the tags should be included as a comma separated arguments to the build command.

How to reproduce it: using sample dag having list tags

    t2 = QuboleOperator(
        task_id='hive_gcs_location',
        command_type="hivecmd",
        script_location="s3n://public-qubole/qbol-library/scripts/show_table.hql",
        notify=True,
        tags=['tag1', 'tag2'],
        trigger_rule="all_done"
    )

and do an airflow run.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:7 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
mik-lajcommented, Sep 14, 2020

@chattarajoy I assigned you to these tickets.

0reactions
chattarajoycommented, Sep 15, 2020

sure @mik-laj, created issues for both the points mentioned will look into it. Could you pls change the assignee for the to me?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Source code for airflow.contrib.hooks.qubole_hook
Source code for airflow.contrib.hooks.qubole_hook ; log = logging ; COMMAND_CLASSES = { ; POSITIONAL_ARGS = { ; def flatten_list( ; def filter_options( ...
Read more >
apache-airflow-providers-qubole 1.0.0rc1 - PyPI
Commit Committed Subject b40dffa08 2020‑12‑08 Rename remaing modules to match AIP‑21 (#12917) 7825e8f59 2020‑11‑13 Docs installation improvements (#12304) b2a28d159 2020‑11‑09 Moves provider packages scripts to dev...
Read more >
Airflow: .../commits.rst | Fossies
This is detailed commit list of changes for versions provider package: qubole . ... [AIRFLOW-9347] Fix QuboleHook unable to add list to tags...
Read more >
Changelog - Apache Airflow Documentation
Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859) ... Add documentation for CLI command airflow dags test (#8251).
Read more >
airflow qubole 源码 - seaxiang
"""Qubole hook""" from __future__ import annotations import datetime import logging ... list], list]: """Build Command argument from command and options""" ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found