[firestore-bigquery-export] Resources exceeded during query execution error

• Extension name: firestore-bigquery-export
• Extension version: 0.1.16
• Configuration values (redact info where appropriate): -Cloud Functions location: us-central1 -BigQuery Dataset location: us

Getting an error selecting from one of my ported collections. all other much larger collections work fine.

Running: select * from firestore_export.mycollection_raw_latest limit 10 Produces: Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak usage: 122% of limit. Top memory consumer(s): sort operations used for analytic OVER() clauses: 98% other/unattributed: 2%

No other query against anything in my db, other firestore ported collections, materialized views, scheduled generated tables from firestore collection ports, is throwing that error.

Workaround: Added my own version of _latest view that I believe is functionally equivalent without the memory pressure:

create view myschema.mycollection_raw_latest as
SELECT document_name, document_id, timestamp, event_id, operation, data
FROM
  (
    SELECT *, ROW_NUMBER() OVER(PARTITION BY document_name ORDER BY timestamp DESC) rn
    FROM firestore_export.mycollection_raw_changelog
  )
WHERE operation<>'DELETE' and rn=1

[meyerovb]

The first query creates a fake changelog table that will reproduce the error, pretend its just a really big document that gets modified very often. The second is your view that goes kaboom. I tried making my own version but you don’t give imports event_ids so nope not happening…

create or replace table changelog_table as with a as ( select DATE_FROM_UNIX_DATE(CAST(RAND() * 10000 AS INT64)) as timestamp, GENERATE_UUID() as event_id, ‘projects/myproject/databases/(default)/documents/mycollection/mydocument’ as document_name, ‘UPDATE’ as operation, string_agg(concat(word, word), ‘’) as data, ‘mydocument’ as document_id from `publicdata.samples.shakespeare` )select a.* from a, `publicdata.samples.shakespeare` limit 20000;

create or replace table your_latest as – Retrieves the latest document change events for all live documents. – timestamp: The Firestore timestamp at which the event took place. – operation: One of INSERT, UPDATE, DELETE, IMPORT. – event_id: The id of the event that triggered the cloud function mirrored the event. – data: A raw JSON payload of the current state of the document. – document_id: The document id as defined in the Firestore database SELECT document_name, document_id, timestamp, event_id, operation, data FROM ( SELECT document_name, document_id, FIRST_VALUE(timestamp) OVER( PARTITION BY document_name ORDER BY timestamp DESC ) AS timestamp, FIRST_VALUE(event_id) OVER( PARTITION BY document_name ORDER BY timestamp DESC ) AS event_id, FIRST_VALUE(operation) OVER( PARTITION BY document_name ORDER BY timestamp DESC ) AS operation, FIRST_VALUE(data) OVER( PARTITION BY document_name ORDER BY timestamp DESC ) AS data, FIRST_VALUE(operation) OVER( PARTITION BY document_name ORDER BY timestamp DESC ) = “DELETE” AS is_deleted FROM changelog_table ORDER BY document_name, timestamp DESC ) WHERE NOT is_deleted GROUP BY document_name, document_id, timestamp, event_id, operation, data;

[cabljac]

Draft PR here https://github.com/firebase/extensions/pull/1288 to fix this.