firestore-bigquery-export: Schema view for maps as dictionaries

[REQUIRED] Step 2: Extension name

This feature request is for extension: firestore-bigquery-export, and in particular for the GENERATE_SCHEMA_VIEWS.md

What feature would you like to see?

In my application, we use dynamic maps (i.e. dictionaries) where we don’t know the keys for the data in advance - they are defined by user-actions in the application (e.g. they might be user-ids). This means we can’t list the keys in the schema.

It would be great to be able to treat a map as a pair of arrays. One for the map’s keys and one for the maps values. That way we can do all the usual great things with bigquery.

How would you use it?

We’d be able to do things like count the number of owners of a keen (we’re using it for staykeen.com) and get summary statistics.

[russellwheatley]

@i14h, I think it might be a good idea to allow a json type in the schema perhaps. This would also cover this feature request: https://github.com/firebase/extensions/issues/408:

Example Schema:

{
  "fields": [
    {
      "name": "map_example",
      "type": "json"
    }
  ]
}

map_example objects taken from Firestore would be stringified and put in a BigQuery column.

[devth]

Would love to see support for this. Until then we can’t use any of fs-bq-schema-views because it doesn’t propagate the raw JSON data for use with BigQuery’s JSON_EXTRACT capabilities.