Does array search support push down?
See original GitHub issueI’m trying to query an array in ElasticSearch
data: "names":[{"name":"allen"},{"name":"bill"},{"name":"dave"},{"name":"poter"}]
goal: "select names from table where array_contains(names.name, "bill")"
but spark won’t do predicate pushdown if SQL statement use array_contains function.
hint: names.name = ["allen","bill","dave","poter"]
I’ve tried
select * from table where array_contains(names.name,"bill")
-- and
select explode(names.name) as name from table as t1;select * from t1 where name = "bill"
-- and
select * from table where cast(names.name as string) like '%bill%'
All failed to do pushdown, any other ways to do it?
Issue Analytics
- State:
- Created 6 years ago
- Comments:8 (4 by maintainers)
Top Results From Across the Web
Array.prototype.push() - JavaScript - MDN Web Docs
The push() method appends values to an array. Array.prototype.unshift() has similar behavior to push() , but applied to the start of an array....
Read more >javascript - Array.push() if does not exist? - Stack Overflow
The '&&' operator means 'and', so if the first condition is true we push it to the array.
Read more >Four Methods to Search Through Arrays in JavaScript
Learn about four approaches to searching for values in arrays: includes, indexOf, find, and filter methods.
Read more >Adding new values to an array using Array.push() replaces all ...
When adding new values to an array object using Array.push(), it replaces all values in the array object with the last value pushed....
Read more >5. Working with Arrays and Loops - JavaScript Cookbook [Book]
One you’ve created an array, using Array object or literal notation, you can access the array elements in a loop, or use any...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
with this change in Spark 3 it might be possible to push down
array_contains
now? https://github.com/apache/spark/pull/28366possibly just add this elasticseach datasource to
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources
?@jbaiera