question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BigQuery INFORMATION_SCHEMA views not parsed properly?

See original GitHub issue

BigQuery has INFORMATION_SCHEMA views that don’t seem to parse properly using sqlglot?

For example: sqlglot.parse_one("SELECT * FROM project.dataset.INFORMATION_SCHEMA.TABLES", read="bigquery") gives:

(SELECT expressions:
  (STAR ), from:
  (FROM expressions:
    (TABLE this:
      (IDENTIFIER this: TABLES, quoted: False), db:
      (IDENTIFIER this: INFORMATION_SCHEMA, quoted: False), catalog:
      (IDENTIFIER this: dataset, quoted: False))))

which leaves out the project part completely. Not quite sure what the correct behavior would be though!

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
barakaloncommented, Sep 26, 2022

I don’t think it makes sense for a column to have a db. Columns select from the tables available in the current scope. They aren’t references to the column itself.

But I think the matter at hand is Table expressions, not Column expressions.

Tables allow for catalog, db, and table. Seems like Bigquery allows for further namespacing, which I don’t think is standard. So we want to gracefully handle this case by nesting the table in an arbitrary number of Dot expressions.

0reactions
GeorgeSittascommented, Sep 26, 2022

Regarding columns, is the following syntax tree what we’re after?

>>> sqlglot.parse_one("SELECT INFORMATION_SCHEMA.TABLES.TABLE_NAME")
(SELECT expressions:
  (DOT this:
    (COLUMN this:
      (IDENTIFIER this: TABLES, quoted: False), table:
      (IDENTIFIER this: INFORMATION_SCHEMA, quoted: False)), expression:
    (IDENTIFIER this: TABLE_NAME, quoted: False)), from:
  (FROM expressions:
    (TABLE this:
      (IDENTIFIER this: TABLES, quoted: False), db:
      (IDENTIFIER this: INFORMATION_SCHEMA, quoted: False))))

The alternative form I had in mind given the context is:

>>> sqlglot.parse_one("SELECT INFORMATION_SCHEMA.TABLES.TABLE_NAME")
(SELECT expressions:
  (COLUMN this:
    (IDENTIFIER this: TABLE_NAME, quoted: False), table:
    (IDENTIFIER this: TABLES, quoted: False), db:
    (IDENTIFIER this: INFORMATION_SCHEMA, quoted: False)), from:
  (FROM expressions:
    (TABLE this:
      (IDENTIFIER this: TABLES, quoted: False), db:
      (IDENTIFIER this: INFORMATION_SCHEMA, quoted: False))))
Read more comments on GitHub >

github_iconTop Results From Across the Web

Getting information about views | BigQuery - Google Cloud
This document describes how to list, get information about, and see metadata for views in BigQuery. You can list views in datasets by:....
Read more >
BigTips: INFORMATION_SCHEMA Views in BigQuery, Part 2 ...
It just pulls from the INFORMATION_SCHEMA. SCHEMATA view to collect the list of datasets, and then iterates through that array to query the ......
Read more >
How to run query after connecting as External Data Source
As part of my attempt to run my first SQL query using BigQuery for a PostgreSQL database, I have been able to connect...
Read more >
Chapter 4. Loading Data into BigQuery - O'Reilly
Could not parse 'NULL' as int for field HBCU (position 26) starting at location ... When presented with the cleaner file, BigQuery correctly...
Read more >
Is there a way to export a BigQuery table's schema as JSON?
The are also interesting metadata in INFORMATION_SCHEMA.VIEWS if you also need the source code from your views. Then hit save results / JSON...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found