Enable Storage API option in BigQuery Connector
See original GitHub issueThis will add a connector provider for BigQuery. This is basically #14790 reloaded
Initially we will use an ODBC driver as in the former implementation, but we plan to use a BQ-specific FDW and use BQ Storage API.
So, the API will made compatible with Storage API and not support arbitrary queries at the moment.
For authorization we’ll use OAuth2 as other connectors (e.g. GDrive). The users will have to connect to BQ using the account page or the import data page.
We’ll add a new service bigquery
to the oauth
configuration in app_config.yml
.
Import job parameters:
connector
parameters:project
name of the BigQuery project
- rest of parameters:
dataset
name of the BQ datasettable
name of the table to importcolumns
(optional; all by default) columns of the table to import (array of strings)filter
(optional) filtering condition in BQ Standard SQL, as a string of the form"column_name operator value"
or other expression involving a single column (see examples below)limit
(optional) maximum number of rows to import
Filter examples:
“int_field > 5”
“date_field = CAST(‘2014-9-27’ as DATE)”
“nullable_field is not NULL”
“st_equals(geo_field, st_geofromtext(“POINT(2, 2)”))”
“numeric_field BETWEEN 1.0 AND 5.0”
I initially considering using the names database
for project
and schema
for dataset
for consitency with other import providers, but I think it will be clear for users to use the BQ terminology.
Issue Analytics
- State:
- Created 4 years ago
- Comments:11 (11 by maintainers)
Top GitHub Comments
🤔 One thing we could do to support arbitrary queries and use storage API is to save the query to a temporary table, then use storage api to transfer it.
Fixed via #15266