Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

OrcWriter may close column writers twice if rollback

See original GitHub issue

During rollback, do not close the writer if the it has already been closed. Need to re-exam outputPageSink’s behavior in compactSorted method.

java.util.concurrent.CompletionException: java.lang.IllegalStateException
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1739)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:488)
        at com.facebook.presto.orc.writer.SliceDictionaryColumnWriter.getBufferedBytes(SliceDictionaryColumnWriter.java:518)
        at com.facebook.presto.orc.DictionaryCompressionOptimizer$DictionaryColumnManager.getBufferedBytes(DictionaryCompressionOptimizer.java:443)
        at com.facebook.presto.orc.DictionaryCompressionOptimizer$DictionaryColumnManager.getCompressionRatio(DictionaryCompressionOptimizer.java:438)
        at com.facebook.presto.orc.DictionaryCompressionOptimizer.convertLowCompressionStreams(DictionaryCompressionOptimizer.java:184)
        at com.facebook.presto.orc.DictionaryCompressionOptimizer.finalOptimize(DictionaryCompressionOptimizer.java:107)
        at com.facebook.presto.orc.OrcWriter.bufferStripeData(OrcWriter.java:368)
        at com.facebook.presto.orc.OrcWriter.flushStripe(OrcWriter.java:332)
        at com.facebook.presto.orc.OrcWriter.close(OrcWriter.java:444)
        at com.facebook.presto.raptor.storage.OrcFileWriter.close(OrcFileWriter.java:149)
        at com.facebook.presto.raptor.storage.OrcStorageManager$OrcStoragePageSink.rollback(OrcStorageManager.java:679)
        at com.facebook.presto.raptor.storage.organization.ShardCompactor.compactSorted(ShardCompactor.java:161)
        at com.facebook.presto.raptor.storage.organization.OrganizationJob.performCompaction(OrganizationJob.java:108)
        at com.facebook.presto.raptor.storage.organization.OrganizationJob.runJob(OrganizationJob.java:83)
        at com.facebook.presto.raptor.storage.organization.OrganizationJob.runJob(OrganizationJob.java:71)
        at com.facebook.presto.raptor.storage.organization.OrganizationJob.run(OrganizationJob.java:59)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
        ... 3 more

Issue Analytics

State:
Created 4 years ago
Comments:8 (5 by maintainers)

Top GitHub Comments

1reaction

golammottcommented, Oct 1, 2019

@highker Sorry for the delay, looking at this now.

0reactions

highkercommented, Oct 1, 2019

@golammott, will you still be available to work on this ticket?

Top Results From Across the Web

WriterImpl (ORC Core 1.8.0 API)

Add arbitrary meta-data to the ORC file. This may be called at any point until the Writer is closed. If the same key...

I/O Nodes - Actian Documentation

Indicates whether parallelism can only be applied on a per-file basis. Typically, reading may be parallelized using subsections (or splits) of a file....

Releases · rapidsai/cudf - GitHub

Adds an EventHandler to Java MemoryBuffer to be invoked on close (#12125) @abellina ... Fix alignment of compressed blocks in ORC writer (#12077)...

Gate Snap - SnapLogic Documentation - Confluence

You can revert to the old UI form by deselecting the New Form UI check box in ... ORC Writer/Reader Snaps fail on...

6. Fixed Issues - Hortonworks Data Platform

BUG-5700, HDFS, NameNode should shut down if out of memory error occurs ... BUG-36111, HIVE-8470, Hive, Orc writer cant handle column of type...