question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Enable Storage API option in BigQuery Connector

See original GitHub issue

This will add a connector provider for BigQuery. This is basically #14790 reloaded

Initially we will use an ODBC driver as in the former implementation, but we plan to use a BQ-specific FDW and use BQ Storage API.

So, the API will made compatible with Storage API and not support arbitrary queries at the moment.

For authorization we’ll use OAuth2 as other connectors (e.g. GDrive). The users will have to connect to BQ using the account page or the import data page. We’ll add a new service bigquery to the oauth configuration in app_config.yml.

Import job parameters:

  • connector parameters:
    • project name of the BigQuery project
  • rest of parameters:
    • dataset name of the BQ dataset
    • table name of the table to import
    • columns (optional; all by default) columns of the table to import (array of strings)
    • filter (optional) filtering condition in BQ Standard SQL, as a string of the form "column_name operator value" or other expression involving a single column (see examples below)
    • limit(optional) maximum number of rows to import

Filter examples:

  • “int_field > 5”
  • “date_field = CAST(‘2014-9-27’ as DATE)”
  • “nullable_field is not NULL”
  • “st_equals(geo_field, st_geofromtext(“POINT(2, 2)”))”
  • “numeric_field BETWEEN 1.0 AND 5.0”

I initially considering using the names database for project and schema for dataset for consitency with other import providers, but I think it will be clear for users to use the BQ terminology.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:11 (11 by maintainers)

github_iconTop GitHub Comments

1reaction
jgoizuetacommented, Oct 28, 2019

🤔 One thing we could do to support arbitrary queries and use storage API is to save the query to a temporary table, then use storage api to transfer it.

0reactions
jgoizuetacommented, Jan 8, 2020

Fixed via #15266

Read more comments on GitHub >

github_iconTop Results From Across the Web

Use the BigQuery Storage Read API to read table data
Enabling the API​​ The Storage Read API is distinct from the BigQuery API, and shows up separately in the Google Cloud console as...
Read more >
How to Enable the Google BigQuery Storage API - MicroStrategy
In the GCP console, go to APIs and services in your project. Search for BigQuery Storage API. Click Enable. Once this feature is...
Read more >
BigQuery Storage API Simplified: A Comprehensive Guide 101
Read APIs allow users to retrieve rows as structured responses with pagination. BigQuery allows creating multiple streams for reading from the ...
Read more >
Set up BigQuery Export - Analytics Help - Google Support
Step 1: Create a Google API Console project and enable BigQuery · Step 2: Prepare your project for BigQuery Export · Step 2.1:...
Read more >
Setting up Google BigQuery as a Target - Rivery Documentation
Google Cloud Service Account; Enable Cloud Storage and BigQuery APIs ... For our last step we will configure a BigQuery connection within Rivery:....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found