BigQueryHook `create_empty_dataset` missing `datasetReference`

Apache Airflow version: 2.0.2

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.19

Environment:

• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
• Kernel (e.g. uname -a): x86_64 GNU/Linux
• Install tools:
• Others:

What happened:

Using the dataset_reference argument for BigQueryCreateEmptyDatasetOperator to set table expiration throws an error:

[2021-05-27 00:03:23,999] {taskinstance.py:1455} ERROR - 'datasetReference'
Traceback (most recent call last):
  File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/google/cloud/operators/bigquery.py", line 1419, in execute
    bq_hook.create_empty_dataset(
  File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/google/common/hooks/base_google.py", line 425, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 414, in create_empty_dataset
    specified_param = dataset_reference["datasetReference"].get(param)
KeyError: 'datasetReference'

What you expected to happen:

I expected no error, the operator shows a similar dataset_reference dict in its documentation:

create_new_dataset = BigQueryCreateEmptyDatasetOperator(                
    dataset_id='new-dataset',                
    project_id='my-project',                
    dataset_reference={"friendlyName": "New Dataset"}, 
    gcp_conn_id='_my_gcp_conn_',                
    task_id='newDatasetCreator',                
    dag=dag
)

How to reproduce it:

create_dataset = BigQueryCreateEmptyDatasetOperator(
    task_id='create_dataset',
    project_id=PROJECT,
    dataset_id=DATASET,
    dataset_reference={"defaultTableExpirationMs": str(1000 * 60 * 60 * 24 * 30)},
    dag=dag
)

Anything else we need to know:

The create_empty_dataset method from the BigQueryHook class expects datasetReference to always be a key in the dictionary:

https://github.com/apache/airflow/blob/86768859c689bf02ced96e71996a3a30da1b5888/airflow/providers/google/cloud/hooks/bigquery.py#L442

I was able to fix the error adding it:

create_dataset = BigQueryCreateEmptyDatasetOperator(
    task_id='create_dataset',
    project_id=PROJECT,
    dataset_id=DATASET,
    dataset_reference={"datasetReference": {}, "defaultTableExpirationMs": str(1000 * 60 * 60 * 24 * 30)},
    dag=dag
)

[rbiegacz]

@eladkal - please, assign this item to me - I will try to fix it.

[eladkal]

@g-saxena assigned