firestore-bigquery-export: Include path parameters as columns directly

This feature request is for extension: firestore-bigquery-export

What feature would you like to see?

When streaming data from firestore to BigQuery, the tables include both the document_name and the document_id. However, data from document_ids at higher levels is not directly accessible and has to parsed from the document_name column. I think it would increase usability if the wilcard parameters could be directly included as separate columns, either directly in the streaming cloud function, or at least in the _latest or _schema views.

Ideally, if one runs npx firebase ext:install firebase/firestore-bigquery-export with COLLECTION_PATH=my-collection/{collection_id}/subcollection, a column named collection_id should be included in the raw_changelog table.

How would you use it?

I am trying to split operational and analytic workloads between firestore and BigQuery. As subcollections as used in firestore to group related data, that information is required for analytics as well, were the entire collection group is merged into one table. Having the grouping information directly available as a column, instead of having to parse it from a string would make things simpler and more robust.

[i14h]

@JarnoRFB we’re thinking of different ways to implements this. We’ll keep you posted.

[dackers86]

Closed via https://github.com/firebase/extensions/pull/891