Query performance 1 minute in Hue Impala vs 2 hours in python fetchall()

In Hue Impala my query runs less than 1 minute, but (exactly) the same query using impyla runs more than 2 hours. The python script runs on the same machine where the Impala daemon runs. My query is a simple “SELECT * FROM my_table WHERE col1 = x;” . The data is (Parquet) partitioned by “col1”.

The performance of the cluster is getting high only on the first half minute after Python script starts. The size of my query result is at about 1GB but the memory usage of my Python script increases continuously from some hundred MB until at about 15GB.

In the Python code cursor.execute(sql_query) finishes in less than 20 seconds (sql_query is the above query), but res = cursor.fetchall() runs for ~2 hours.

Does anybody know what the problem could be? Thank you!

[schaffino]

@hikoz identified issue with NULL values I inadvertently caused due to indexing from the back of the null bit array and some additional padding that is added by the fix for

HUE-2722, HiveServer2 sometimes does not add trailing

Have just changed to index from the front to avoid the issue and submitted above pull request.

The query below previously returning incorrectly now works.

select stack(2, NULL, NULL, NULL,1.1,‘1’,1) as (a,b,c)

[schaffino]

I’m not particularly familiar with anaconda but I would try a conda uninstall and then pip uninstall of impyla as it can remove leftovers from mixes of manual installs alongside packaged installs. yOu can also modify setup.py to have a version you can easily identify with pip.

In terms of your memory issues, sounds like the date objects will be taking a fair amount of space. Have you tried the convert_types=False option when creating the cursor?

You can download guppy and run the below after pulling the dataset to see what object types are taking the most memory.

from guppy import hpy
h = hpy()
print h.heap()

Native python object types and data structures aren’t particularly memory efficient. What are you wanting to do with the whole dataset in memory? You could look at going directly a more compact in memory format.