[sdk] failure launching Dataproc jobs

Environment

• KFP version: Vertex AI
• KFP SDK version: 1.7
• All dependencies version: kfp 1.8.2 kfp-pipeline-spec 0.1.11 kfp-server-api 1.7.0

Steps to reproduce

@dsl.pipeline(
  name='pipeline',
  pipeline_root='gs://bucket'
)
def pipeline():
  dataproc_submit_pyspark_op(
    project_id=project,
    region=region,
    cluster_name=cluster_name,
    main_python_file_uri='gs://bucket/valid_job.py',
  )

Expected result

submission of a dataproc job.

Materials and Reference

error from vertex runner:

 File "/ml/kfp_component/google/dataproc/_submit_job.py", line 49, in submit_job "
Error
2021-10-05 02:41:50.279 PDT
workerpool0-0
" client = DataprocClient() "
Error
2021-10-05 02:41:50.279 PDT
workerpool0-0
" File "/ml/kfp_component/google/common/_utils.py", line 170, in __init__ "
Error
2021-10-05 02:41:50.279 PDT
workerpool0-0
" self._build_client() "
Error
2021-10-05 02:41:50.279 PDT
workerpool0-0
"TypeError: _build_client() takes 0 positional arguments but 1 was given"

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[montenegrodr]

Falling back to gcr.io/ml-pipeline/ml-pipeline-gcp:1.6.0 worked.

[chensun]

Fixed by https://github.com/kubeflow/pipelines/pull/7364