question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

firestore-bigquery-export fs-bq-import-collection doesn't allow {widecard} path

See original GitHub issue

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

  • For general technical questions, post a question on StackOverflow with the firebase tag.
  • For general Firebase discussion, use the firebase-talk google group.
  • To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

  • Extension name: firestore-bigquery-export
  • Configuration values (redact info where appropriate):
    • fs-bq-import-collection

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

  1. Clone this repo.
  2. Go to extensions/firestore-bigquery-export/scripts/import and install dependencies (npm install)
  3. Execute npm run import
  4. Pass the ${PROJECT_ID}
  5. Try pass a widecard path as described on docs, like chats/{chatid}/posts
  6. Get an error: >> The collection path must only contain letters or spaces
Expected result

Allow widecard paths to import.

Actual result

Get an error: >> The collection path must only contain letters or spaces

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:23
  • Comments:24 (5 by maintainers)

github_iconTop GitHub Comments

10reactions
MattReidArnoldcommented, Dec 23, 2019

@kienner-philippe and @garciapd, I was able to get around this by making my own copy of the script and modifying the query to use collectionGroups.

Changed this: https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/scripts/import/src/index.ts#L188

To this:

    let query = firebase
      .firestore()
      .collectionGroup(sourceCollectionPath)
      .limit(batch);

Added these to my package.json:

"devDependencies": {
    "@firebaseextensions/firestore-bigquery-change-tracker": "^1.0.0",
    "@google-cloud/bigquery": "^4.4.0",
    "inquirer": "^7.0.0",
}

Then I just run my local version of the script using ts-node with something like this:

GOOGLE_APPLICATION_CREDENTIALS="./secrets/service-account-key.json" \
    npx ts-node ./bin/firestore-bigquery-export-backfill.ts  # my local version of the file

when the script asks:

What is the path of the the Cloud Firestore Collection you would like to import from?

Give it the sub-collection name for the collection group query. posts

7reactions
liberathorcommented, Sep 16, 2020

please support sub collections!!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Loading data from Firestore exports | BigQuery - Google Cloud
BigQuery supports loading data from Firestore exports created using the Firestore managed import and export service. The managed import and export service ...
Read more >
How to configure the “Export Collections to BigQuery ...
Let's use the “QUERY” button at the top to query this BigTable view. A SELECT * FROM SQL query returns two rows, corresponding...
Read more >
Existing firestore collection data into bigQuery - Stack Overflow
This extension only sends the content of documents that have been changed -- it does not export your full dataset of existing documents...
Read more >
Re: Firestore -> BigQuery - Google Cloud Community
The process of exporting your data and updates in real-time is called streaming and luckily, there is an extension that helps you stream...
Read more >
Stream Collections to BigQuery - Extensions Marketplace
Use this extension to export the documents in a Cloud Firestore collection to ... it does not export your full dataset of existing...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found