Parameter optional_components =True don’t create dataproc cluster with web interface(component gateway)

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

Google Cloud Composer-2.0.0-preview.5 Airflow-2.1.4

Apache Airflow version

2.1.4

Operating System

UNIX

Deployment

Composer

Deployment details

No response

What happened

I write this dag(above) in cloud composer to create and delete dataproc cluster. But when the cluster is created even with the option enable_component_gateway=True it does not enable the component gateway with access to jupyter notebook as parameterized in the dag. But the additional components are enabled as per the image.

`from airflow.contrib.sensors.gcs_sensor import GoogleCloudStoragePrefixSensor from airflow import DAG from datetime import datetime, timedelta from airflow.contrib.operators.gcs_to_bq import GoogleCloudStorageToBigQueryOperator from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator, DataprocDeleteClusterOperator, ClusterGenerator

yesterday = datetime.combine(datetime.today() - timedelta(1), datetime.min.time())

default_args = { ‘owner’: ‘teste3’, ‘depends_on_past’: False, ‘start_date’ :yesterday, ‘email’: [‘[e-mail]’], ‘email_on_failure’: False, ‘email_on_retry’: False, ‘retries’: 0, ‘retry_delay’: timedelta(minutes=5),

}

dag = DAG( ‘teste-dag-3’,catchup=False, default_args=default_args, schedule_interval=None)

CLUSTER_GENERATOR = ClusterGenerator( project_id=“sandbox-coe”, cluster_name=‘teste-ge-{{ ds }}’, num_masters=1, master_machine_type=‘n2-standard-8’, worker_machine_type=‘n2-standard-8’, worker_disk_size=500, master_disk_size=500, master_disk_type=‘pd-ssd’, worker_disk_type=‘pd-ssd’, image_version=‘1.5.56-ubuntu18’, tags=[‘allow-dataproc-internal’], region=‘us-central1’, zone=‘us-central1-f’, storage_bucket = ‘bucket-dataproc-ge’, labels = {‘product’ : ‘sample-label’}, enable_component_gateway=True, # this is not working optional_components = [ ‘JUPYTER’, ‘ANACONDA’ ] ).make()

create_cluster=DataprocCreateClusterOperator( dag=dag, task_id=‘start_cluster_example’, cluster_name=‘teste-ge-{{ ds }}’, project_id=“sandbox-coe”, cluster_config=CLUSTER_GENERATOR, region=‘us-central1’ )

stop_cluster_example = DataprocDeleteClusterOperator( dag=dag, task_id=‘stop_cluster_example’, cluster_name=‘teste-ge-{{ ds }}’, project_id= ‘sandbox-coe’, region=‘us-central1’, ) #stops a running dataproc cluster

create_cluster >> stop_cluster_example `

What you expected to happen

I hope it happens that when I create the cluster, the web interface components that I activated when creating the cluster appear as in the image

How to reproduce

Execute the dag above in cloud composer with DataprocCreateClusterOperator.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[aoelvp94]

Yeah the new version works for me. Thanks for the update @ThiagoPositeli !

[aoelvp94]

🙏 🙏 same problem here 😢