question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Does array search support push down?

See original GitHub issue

I’m trying to query an array in ElasticSearch

data: "names":[{"name":"allen"},{"name":"bill"},{"name":"dave"},{"name":"poter"}]
goal: "select names from table where array_contains(names.name, "bill")"
but spark won’t do predicate pushdown if SQL statement use array_contains function. hint: names.name = ["allen","bill","dave","poter"] I’ve tried

select * from table where array_contains(names.name,"bill") -- and select explode(names.name) as name from table as t1;select * from t1 where name = "bill" -- and select * from table where cast(names.name as string) like '%bill%' All failed to do pushdown, any other ways to do it?

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
Tagarcommented, Jun 14, 2020

with this change in Spark 3 it might be possible to push down array_contains now? https://github.com/apache/spark/pull/28366

possibly just add this elasticseach datasource to spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources ?

0reactions
kant777commented, Jun 17, 2020
Read more comments on GitHub >

github_iconTop Results From Across the Web

Array.prototype.push() - JavaScript - MDN Web Docs
The push() method appends values to an array. Array.prototype.unshift() has similar behavior to push() , but applied to the start of an array....
Read more >
javascript - Array.push() if does not exist? - Stack Overflow
The '&&' operator means 'and', so if the first condition is true we push it to the array.
Read more >
Four Methods to Search Through Arrays in JavaScript
Learn about four approaches to searching for values in arrays: includes, indexOf, find, and filter methods.
Read more >
Adding new values to an array using Array.push() replaces all ...
When adding new values to an array object using Array.push(), it replaces all values in the array object with the last value pushed....
Read more >
5. Working with Arrays and Loops - JavaScript Cookbook [Book]
One you’ve created an array, using Array object or literal notation, you can access the array elements in a loop, or use any...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found