Reading only requested number of partitions from BQ table doesn't work

I am trying to query table which is partitioned by date field. Like this,

val prog_logs = spark.read.format("bigquery")
        .option("table", "project1:dataset.table")
        .option("filter", " date between '2019-09-10' and  '2019-09-11' ")
        .load()
        .cache()

This is reading entire table instead of only ‘2019-09-10’ and ‘2019-09-11’ partitions.

[emkornfield]

Sorry for the late response but for proper push down in spark 2.4 I think you need: date between date(‘2019-09-10’) and date(‘2019-09-11’) (i.e. explicitly cast values to date).

[davidrabinowitz]

@jesuejunior is it still an issue? Can you please share the log of a sample app (it shows the pushed down filters)