DataProcOperator: TypeError: upload() got an unexpected keyword argument 'bucket_name'

Some modules version that could help:

apache-airflow==1.10.10 google-cloud-storage==1.26.0

Logs

[2020-07-12 08:03:03,455] {taskinstance.py:1145} ERROR - upload() got an unexpected keyword argument 'bucket_name'
Traceback (most recent call last):
  File "/airflow/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/airflow/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/contrib/operators/dataproc_operator.py", line 1145, in execute
    self.main = self._upload_file_temp(bucket, self.main)
  File "/airflow/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/contrib/operators/dataproc_operator.py", line 1108, in _upload_file_temp
    filename=local_file
TypeError: upload() got an unexpected keyword argument 'bucket_name'
[2020-07-12 08:03:03,457] {taskinstance.py:1202} INFO - Marking task as FAILED.dag_id=snowplow_pipeline, task_id=text_to_partitioned_parquet, execution_date=20200711T000000, start_date=20200712T000301, end_date=20200712T000303
[2020-07-12 08:03:11,950] {logging_mixin.py:112} INFO - [2020-07-12 08:03:11,949] {local_task_job.py:103} INFO - Task exited with return code 1

Potential fix

https://github.com/apache/airflow/blob/96697180d79bfc90f6964a8e99f9dd441789177c/airflow/contrib/operators/dataproc_operator.py#L1105

Remove the name postfix from the bucket_name and object_name.

GoogleCloudStorageHook(
1103             google_cloud_storage_conn_id=self.gcp_conn_id
1104         ).upload(
1105             bucket=bucket,
1106             object=temp_filename,
1107             mime_type='application/x-python',
1108             filename=local_file
1109         )

[mik-laj]

Do I need to modify my current DAG file in order for backport to work (such import statements) or special commands to clear anything like database and etc?

You must start using the new import paths. This allows you to use new and old operators to be able to migrate gradually… Please look at: https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/dlp.py Google will discuss in detail the migration to new operators at the Airflow Summit conference. https://airflowsummit.org/sessions/migration-to-airflow-backport-providers/

[github-actions[bot]]

This issue has been closed because it has not received response from the issue author.