Add MERGE functionality

In BigQuery it is possible to merge two tables:

MERGE DATASET.table T
USING (SELECT * FROM DATASET.another_table) S
ON T.id = S.id
WHEN MATCHED THEN
	UPDATE SET value = S.value
WHEN NOT MATCHED THEN
	INSERT ROW

Is it possible to add such functionality to DataFrame.to_gbq function? I.e. something like df.to_gbq(if_exists='merge', on_a='id', on_b='id'), or even provide a possibilty to write a whole ON clause df.to_gbq(if_exists='merge', on='A.city = B.city AND A.date = B.date'

[parthea]

@violetbrina , PRs are welcome!

[ZiggerZZ]

I’m not yet familiar with pandas-gbq codebase, so it will take me a while. In my projects, I split data into batches and merge them into the main using temporary tables. Can we just do like this? Here’s the psedocode:

# we want to merge `data` to `table_id` using `condition`
data_batches = split_data_into_batches(data)
for data_batch in data_batches:
    tmp_id = _id_generator()
    table_tmp_id = f"{table_id}_tmp_{tmp_id}"
    client.create_table(table=bigquery.Table(table_tmp_id, schema=table_id.schema)), expiration=1 hour)
    query = f"""MERGE `{table_id}` T
USING `{table_tmp_id}` S
ON {condition}
WHEN MATCHED THEN
	UPDATE SET ...
WHEN NOT MATCHED THEN
	INSERT ROW"""
    client.query(query).result()