Trying to load data from BigQuery to Spark Dataframe using query option

When i execute the following code, I expect the filter pushdown to work.

reader = self.spark.read.format("bigquery") \
                .option('parentProject', self.project_id) \
                .option('viewsEnabled', True) \
                .option("materializationDataset", "tmp") \
                .option("query", f"select * from `{table_id}`")\
                .option("filter", "CAST(`_TABLE_SUFFIX` AS STRING) = '20170405'")

But instead I get an error Caused by: com.google.cloud.spark.bigquery.repackaged.io.grpc.StatusRuntimeException: INVALID_ARGUMENT: request failed: Row filter for bigquery-rollup.tmp._bqc_c372ae3bcf3a46019600500ebacdf15d is invalid. Filter is ‘(CAST(_TABLE_SUFFIX AS STRING) = ‘20170405’)’

When I run the same without operation without “query” option, it works

reader = self.spark.read \
                .format('bigquery') \
                .option('parentProject', self.project_id) \
                .option('dataset', self.dataset_id) \
                .option('table', self.table_id) \
                .option('filter', "CAST(`_TABLE_SUFFIX` AS STRING) = '20170405'")

Don’t you have support for filters in query based read? Please suggest. Thanks.

[davidrabinowitz]

My guess is that you are reading from a table with a dated suffix as way of partitioning. When loading from query you actually run the query on BigQuery and the materialized result is save in a temporary table (bigquery-rollup.tmp._bqc_c372ae3bcf3a46019600500ebacdf15d in our case), which you cannot use the filter on it.

As an alternative, you can combine the query and the filter:

reader = self.spark.read.format("bigquery") \
                .option('parentProject', self.project_id) \
                .option('viewsEnabled', True) \
                .option("materializationDataset", "tmp") \
                .load(f"SELECT * FROM `{table_id}` WHERE CAST(`_TABLE_SUFFIX` AS STRING) = '20170405'")

[akashgangulyhf]

Thanks for all the info. Was really very helpful. Have a good one.