Pushdown doesn't work with nested fields

Elasticsearch & ES-Hadoop 5.0.0-beta1, Spark 2.0.0

curl -XPOST localhost:9200/pushdown/pushdown -d '{"a":"b","c":{"d":"e"}}'

df = sqlContext.read.format("org.elasticsearch.spark.sql").load("pushdown/pushdown")
df.filter(df.a == "b").show()

as expected generates:

{"query":{"bool":{"must":[{"match_all":{}}],"filter":[{"exists":{"field":"a"}},{"match":{"a":"b"}}]}}}

df.filter(df.c.d == "e").show()

doesn’t generate any pushdown:

{"query":{"match_all":{}}}

[ebuildy]

I believe this is happen also for column/field pruning, and it looks a complicate issue, with the latest version (6.4.0), I see the root field is sent, but never the whole nested field.

If we look the code source: https://github.com/elastic/elasticsearch-hadoop/blob/master/spark/sql-20/src/main/scala/org/elasticsearch/spark/sql/DefaultSource.scala#L233 this plugin “forward” the column/field given by Spark to the search query builder, so the problem is not here.

Also, look like they (Spark team) have just done this for Parquet file format: https://github.com/apache/spark/pull/21320/files

[masseyke]

If I’m reading this correctly, spark does not allow predicates for nested fields to be pushed down for non-hadoop backends in DSv1: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala#L110. If that’s the case, we need #1801 before we can do this.