question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

DynamicFiltering throws prestoException due to Invalid partition value 'N'

See original GitHub issue

When using Hive connector to fetch data that has null value in partitioning key, query fails as follows

com.facebook.presto.spi.PrestoException: Invalid partition value '\N' for BIGINT partition key: ss_sold_date_sk at com.facebook.presto.hive.HiveUtil.bigintPartitionKey(HiveUtil.java:710) at com.facebook.presto.hive.HiveUtil.parsePartitionValue(HiveUtil.java:596) at com.facebook.presto.hive.HivePageSourceProvider.shouldSkipPartition(HivePageSourceProvider.java:547) at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:172) at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:63) at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:80) at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:231) at com.facebook.presto.operator.Driver.processInternal(Driver.java:417) at com.facebook.presto.operator.Driver.lambda$processFor$8(Driver.java:300) at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:721) at com.facebook.presto.operator.Driver.processFor(Driver.java:293) at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077) at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162) at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:545) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
kewang1024commented, Nov 18, 2020

there’s glitch in handling partition pruning logic when hive partitioning key contains a null value. I can try to fix it

Thanks for finding the issue, yeah, it would be great if you can help work on it, you can add me as the reviewer when you have the fix, thanks!

1reaction
highkercommented, Nov 17, 2020
Read more comments on GitHub >

github_iconTop Results From Across the Web

Dynamic filtering — Trino 403 Documentation
When dynamic filtering is enabled, Trino collects candidate values for join ... dynamic filters are used to skip loading of partitions which don't...
Read more >
Unable to load data in Hive partitioned table - Stack Overflow
If you want to load data into a Hive partition, you have to provide the value of the partition itself in the LOAD...
Read more >
[#HIVE-7855] Invalid partition values specified when writing to ...
This exception is thrown because the partition values list is empty while the partition keys list is not. It seems to me that...
Read more >
Load Data throws error for partitioned table.
Query: load data inpath '/user/impala/data/id.txt' into table tbl_id_part partition(year=2010) ERROR: AnalysisException: Partition spec does ...
Read more >
大数据常见错误解决方案 - CSDN博客
9、Reconnect due to socket error: java.nio.channels.ClosedChannelException. 解决方法:kafka服务器broker ip写对. 10、java.lang.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found