question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

The apache-airflow-backport-providers-ssh version 2020.10.29 is broken

See original GitHub issue

Apache Airflow version: 1.10.14 ( 1.10.14+composer)

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.16.15-gke.6000

Environment:

  • Cloud Composer

What happened:

The apache_airflow_backport_providers_ssh-2020.10.29-py3-none-any.whl has got extra files in it which get installed and breaks the use of some hooks and operators. For example, it brings a version of airflow/providers/google/cloud/hooks/compute.py that is older than the one I have and need in my environment: I have the backported providers version 2020.11.23 and this package brings providers’ files of version 2020.10.29.

What you expected to happen:

I expect the wheel files to basically contain two files (plus or minus the init files), just like the tar file. Namely: airflow/providers/ssh/hooks/ssh.py and airflow/providers/ssh/operators/ssh.py but it actually contains 786 files (I truncated the list to the top folders):

apache_airflow_backport_providers_ssh-2020.10.29-py3-none-any > ls -lR
total 0
drwxr-xr-x@ 4 bchazalet  staff  128 27 Jan 11:57 airflow
drwxr-xr-x@ 8 bchazalet  staff  256 27 Jan 11:57 apache_airflow_backport_providers_ssh-2020.10.29.dist-info

./airflow:
total 0
drwxr-xr-x@ 47 bchazalet  staff  1504 27 Jan 11:58 providers

./airflow/providers:
total 0
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 amazon
drwxr-xr-x@ 12 bchazalet  staff  384 27 Jan 11:57 apache
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 celery
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 cloudant
drwxr-xr-x@  3 bchazalet  staff   96 27 Jan 11:57 cncf
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 databricks
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 datadog
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 dingding
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 discord
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 docker
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 elasticsearch
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 exasol
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 facebook
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 ftp
drwxr-xr-x@ 11 bchazalet  staff  352 27 Jan 11:58 google
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 grpc
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 hashicorp
drwxr-xr-x@  8 bchazalet  staff  256 27 Jan 11:57 http
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 imap
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 jdbc
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 jenkins
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 jira
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 microsoft
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 mongo
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 mysql
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 odbc
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 openfaas
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 opsgenie
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 oracle
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 pagerduty
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 plexus
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 postgres
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 presto
drwxr-xr-x@  8 bchazalet  staff  256 27 Jan 11:57 qubole
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 redis
drwxr-xr-x@  8 bchazalet  staff  256 27 Jan 11:57 salesforce
drwxr-xr-x@  5 bchazalet  staff  160 27 Jan 11:57 samba
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 segment
drwxr-xr-x@  7 bchazalet  staff  224 27 Jan 11:57 sftp
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 singularity
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 slack
drwxr-xr-x@  8 bchazalet  staff  256 27 Jan 11:57 snowflake
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 sqlite
drwxr-xr-x@  6 bchazalet  staff  192 27 Jan 11:57 ssh

I think the build process of the wheel file went wrong and included lots of files that should not be there.

How to reproduce it:

You can either manually download the wheel file or simply install apache-airflow-backport-providers-ssh and look at all the files it installs.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
bchazaletcommented, Jan 27, 2021

FYI using ==2020.10.5 works just fine for me.

0reactions
potiukcommented, Feb 25, 2021

Fixed with 2021.2.5 release

Read more comments on GitHub >

github_iconTop Results From Across the Web

apache-airflow-backport-providers-ssh 2021.3.3 - PyPI
Provider classes summary. In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the ssh provider are in the airflow.providers.ssh package.
Read more >
Package apache-airflow-providers-ssh
This is detailed commit list of changes for versions provider package: ssh . For high-level changelog, see package information including changelog. 2.4.1¶.
Read more >
apache-airflow-backport-providers-ssh - Python package - Snyk
A security vulnerability was detected in an indirect dependency that is added to your project when the latest version of apache-airflow-backport ...
Read more >
How we can use SFTPToGCSOperator in GCP composer ...
apache -airflow-backport-providers-ssh==2020.10.29 ... You can keep an eye on Cloud Composer version lists in order to see when the newest ...
Read more >
Import operators from backport provider packages
Install apache-airflow-backport-providers-google and specify the [salesforce] extra in the Extras and version field. Note: To see a list of cross provider ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found