firestore-bigquery-export: Schema view for maps as dictionaries
See original GitHub issue[REQUIRED] Step 2: Extension name
This feature request is for extension: firestore-bigquery-export
, and in particular for the GENERATE_SCHEMA_VIEWS.md
What feature would you like to see?
In my application, we use dynamic maps (i.e. dictionaries) where we don’t know the keys for the data in advance - they are defined by user-actions in the application (e.g. they might be user-ids). This means we can’t list the keys in the schema.
It would be great to be able to treat a map as a pair of arrays. One for the map’s keys and one for the maps values. That way we can do all the usual great things with bigquery.
How would you use it?
We’d be able to do things like count the number of owners of a keen (we’re using it for staykeen.com) and get summary statistics.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:5
- Comments:10 (3 by maintainers)
Top Results From Across the Web
Loading data from Firestore exports | BigQuery - Google Cloud
When you load data into BigQuery from a Firestore export, note the following restrictions ... The schema is inferred for a Firestore export....
Read more >Supported data types | Firestore - Firebase
This page describes the data types that Cloud Firestore supports. ... Map fields are sorted by key and compared by key-value pairs, first...
Read more >[UA] BigQuery Export schema - Analytics Help - Google Support
Field Name Data Type Description
fullVisitorId STRING The unique visitor ID.
userId STRING Overridden User ID sent to Analytics.
visitStartTime INTEGER The timestamp (expressed as POSIX...
Read more >BigQuery: Import of Cloud Firestore export treats map fields as ...
Recently, schema auto-detection has begun failing for these imports, resulting in Firestore map fields being treated as unqueryable byte fields ...
Read more >Collibra Data Dictionary integration - Hackolade
Check for the proper configuration · 1) confirm that the out-of-the-box assetTypes exist: schema, table, database view, column, foreign key, mapping ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@i14h, I think it might be a good idea to allow a
json
type in the schema perhaps. This would also cover this feature request: https://github.com/firebase/extensions/issues/408:Example Schema:
map_example
objects taken from Firestore would be stringified and put in a BigQuery column.Would love to see support for this. Until then we can’t use any of
fs-bq-schema-views
because it doesn’t propagate the raw JSONdata
for use with BigQuery’sJSON_EXTRACT
capabilities.