Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

apache-airflow-providers-google incompatible with latest google-cloud-dataproc (2.6.0)

See original GitHub issue

Apache Airflow version

2.1.4 (latest released)

Operating System

MacOS

Versions of Apache Airflow Providers

apache-airflow-providers-google==5.1.0

Deployment

Other

Deployment details

No response

What happened

Installing the google provider and importing the DataprocSubmitJobOperator fails.

I believe this is due to the removal of the Dataproc V1Beta2 Client in https://github.com/googleapis/python-dataproc/pull/253.

How to reproduce

From a fresh virtualenv:

> pip install apache-airflow-providers-google
> python
Python 3.7.3 (default, Jan 24 2020, 16:24:47)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from airflow.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperator
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/samwheating/Desktop/tmpairflow/lib/python3.7/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 33, in <module>
    from google.cloud.dataproc_v1beta2 import Cluster
ModuleNotFoundError: No module named 'google.cloud.dataproc_v1beta2'

Anything else

Its easy enough to pin the google-cloud-dataproc requirement to an older version, but should we also refactor the dataproc operators to stop using the deprecated client?

Either way I can help out with a PR.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project’s Code of Conduct

Issue Analytics

State:
Created 2 years ago
Comments:9 (9 by maintainers)

Top GitHub Comments

1reaction

mik-lajcommented, Sep 25, 2021

Latest release has been YANKED, so it is not problem anymore. Screenshot_20210925-140844 https://pypi.org/project/google-cloud-dataproc/#history

0reactions

lwyszomicommented, Oct 11, 2021

@SamWheating @kaxil @mik-laj @uranusjr

FYI migration to v1 and upgrade to 3.0.0 already sent to review.

Top Results From Across the Web

apache-airflow-providers-google uses deprecated ... - GitHub

Apache Airflow Provider(s) google Versions of Apache Airflow Providers ... The latest version of the Google Ads API supported by this is V8 ......

apache-airflow-providers-google Documentation

All classes for this provider package are in airflow.providers.google python ... airflow 2.1+ installation via pip install apache-airflow-providers-google ...

[GitHub] [airflow] SamWheating opened a new issue #18485

... [airflow] SamWheating opened a new issue #18485: apache-airflow-providers-google incompatible with latest google-cloud-dataproc (2.6.0).

Troubleshooting PyPI package installation | Cloud Composer

In the build log, find the error message from pip . For example: ERROR: apache-airflow-backport-providers-google 2021.2.5 has ...

pzthon requirement.parse('protobuf<4.0.0dev,>=3.19.0 ...

Protocol Buffer Basics: Python | Protocol Buffers | Google Developers ... google-cloud-dataproc-metastore==1.6.3->apache-airflow-providers-google==6.8.0->-r ...