[firestore-bigquery-export] not creating Ingestion-time partitioned table

Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.1.13
• Configuration values:-
  • Cloud functions location: eu-west1
  • Big Query Dataset location: eu
  • Collection path: sales
  • Dataset ID: firestore-sales-stream
  • BigQuery SQL table partitioning option: DAY

Describe the problem

According to the documentation here I should be able to run the following query to get a list of partitions (admittedly I’ve only set this up today and back filled the table with the supplied script so I’d only expect to see 1 partition):-

SELECT _PARTITIONDATE as pt, FORMAT_TIMESTAMP("%Y%m%d", _PARTITIONDATE) as partition_date
FROM `****.firestore_sales_stream.sales_raw_changelog`
GROUP BY _PARTITIONDATE
ORDER BY _PARTITIONDATE

Steps to reproduce:

Install, configure and run the extension from within the Firebase Dashboard for my project.

Back fill the table with the script from the instructions specified here

Expected result

A (short) list of formatted partition dates based on the pseudo columns _PARTITIONTIME and _PARTITIONDATE

Actual result

The BiqQuery console reports the following error:-

Error running query
Unrecognized name: _PARTITIONDATE; Did you mean _PARTITION_DATE? 

Additional

I’m just trying to get my head around querying partitioned tables to improve performance and seem to be stumbling at the first hurdle.

I can confirm that all my data was successfully back filled and changes to documents in my collection are being streamed fine. Just curious as to why I don’t seem to have a partitioned table?

Thanks!

Ben

[soarb]

That’s the problem really, I can’t list/find/query any columns like this from the table generated by the extension. Here’s what the schema looks like:-

All the expected fields are present apart from the pseudo columns for an ingestion-time partitioned table. As the docs state:-

When you create an ingestion-time partitioned table, two pseudo columns are added to the table: a _PARTITIONTIME pseudo column and a _PARTITIONDATE pseudo column

I was hoping to query for these columns to prove that the partitioning was working, then update our sales report query so that it only targeted a subset of the last 7 days rather than the entire table.

At the moment, the query takes a significant time to run and of course will continue to take longer and longer as new sales data is streamed.

[dackers86]

Based on the config provided. I’ll add this to our tracker as a potential bug