[BUG] IndexOutOfBoundsException during read from input stream

Describe the bug Under unknown conditions reading from InputStream, opened on existing blob, causes IndexOutOfBoundsException in ByteBuffer#checkBounds.

Exception or Stack Trace

checkBounds:567, Buffer (java.nio)
get:149, HeapByteBuffer (java.nio)
readInternal:329, StorageInputStream (com.azure.storage.common)
read:289, StorageInputStream (com.azure.storage.common)
readFully:24, FileInfoAsSeekable (org.jetbrains.hdfsplugin.hdfs.localcache.orcsupport)
readFully:111, FSDataInputStream (org.apache.hadoop.fs)
readRanges:99, JetbrainsOrcDataReader (org.jetbrains.hdfsplugin.hdfs.localcache.orcsupport)
readFileData:62, JetbrainsOrcDataReader (org.jetbrains.hdfsplugin.hdfs.localcache.orcsupport)
readData:151, StripePlanner (org.apache.orc.impl.reader)
readStripe:190, JetbrainsOrcRecordReader (org.jetbrains.hdfsplugin.hdfs.localcache.orcsupport)
advanceStripe:159, JetbrainsOrcRecordReader (org.jetbrains.hdfsplugin.hdfs.localcache.orcsupport)
advanceToNextRow:137, JetbrainsOrcRecordReader (org.jetbrains.hdfsplugin.hdfs.localcache.orcsupport)
<init>:89, JetbrainsOrcRecordReader (org.jetbrains.hdfsplugin.hdfs.localcache.orcsupport)
readBatch:130, ORCContentDownloader (org.jetbrains.hdfsplugin.hdfs.localcache)
writeCsv:43, ORCContentDownloader (org.jetbrains.hdfsplugin.hdfs.localcache)
download:38, ORCContentDownloader (org.jetbrains.hdfsplugin.hdfs.localcache)
compute:86, RfsFileContentManager$downloadFilePart$1 (com.jetbrains.bigdatatools.rfs.localcache)
compute:17, RfsFileContentManager$downloadFilePart$1 (com.jetbrains.bigdatatools.rfs.localcache)
lambda$runProcess$3:179, CoreProgressManager (com.intellij.openapi.progress.impl)
run:-1, 1806785439 (com.intellij.openapi.progress.impl.CoreProgressManager$$Lambda$878)
lambda$runProcess$2:163, CoreProgressManager (com.intellij.openapi.progress.impl)
run:-1, 115725296 (com.intellij.openapi.progress.impl.CoreProgressManager$$Lambda$240)
registerIndicatorAndRun:585, CoreProgressManager (com.intellij.openapi.progress.impl)
executeProcessUnderProgress:531, CoreProgressManager (com.intellij.openapi.progress.impl)
executeProcessUnderProgress:59, ProgressManagerImpl (com.intellij.openapi.progress.impl)
runProcess:150, CoreProgressManager (com.intellij.openapi.progress.impl)
runProcess:179, CoreProgressManager (com.intellij.openapi.progress.impl)
downloadFilePart:85, RfsFileContentManager (com.jetbrains.bigdatatools.rfs.localcache)
getOrDownload:64, RfsFileContentManager (com.jetbrains.bigdatatools.rfs.localcache)
getOrDownload$default:62, RfsFileContentManager (com.jetbrains.bigdatatools.rfs.localcache)
getContentFile:50, RfsFileContentManager (com.jetbrains.bigdatatools.rfs.localcache)
getContentFile$default:49, RfsFileContentManager (com.jetbrains.bigdatatools.rfs.localcache)
loadContent:106, GeneralContentTypeViewerBase (com.jetbrains.bigdatatools.rfs.view)
invoke:76, RfsAsyncFileTypeViewer$openViewer$$inlined$let$lambda$1 (com.jetbrains.bigdatatools.rfs.view)
invoke:71, RfsAsyncFileTypeViewer$openViewer$$inlined$let$lambda$1 (com.jetbrains.bigdatatools.rfs.view)
run:17, ThreadUtilsKt$executeOnPooledThread$1 (com.jetbrains.bigdatatools.util)
run:238, ApplicationImpl$1 (com.intellij.openapi.application.impl)
call:511, Executors$RunnableAdapter (java.util.concurrent)
run$$$capture:266, FutureTask (java.util.concurrent)
run:-1, FutureTask (java.util.concurrent)
 - Async stack trace
<init>:151, FutureTask (java.util.concurrent)
newTaskFor:87, AbstractExecutorService (java.util.concurrent)
submit:111, AbstractExecutorService (java.util.concurrent)
executeOnPooledThread:220, ApplicationImpl (com.intellij.openapi.application.impl)
executeOnPooledThread:16, ThreadUtilsKt (com.jetbrains.bigdatatools.util)
openViewer:74, RfsAsyncFileTypeViewer (com.jetbrains.bigdatatools.rfs.view)
openViewer:34, FileTypeViewerManager (com.jetbrains.bigdatatools.rfs.view)
openViewer$default:33, FileTypeViewerManager (com.jetbrains.bigdatatools.rfs.view)
showContent:18, ShowFileContentAction$Companion (com.jetbrains.bigdatatools.rfs.projectview.actions)
navigate:15, RfsFileNode (com.jetbrains.bigdatatools.rfs.projectview.nodes)
navigateToSource:119, OpenSourceUtil (com.intellij.util)
navigate:74, OpenSourceUtil (com.intellij.util)
navigate:58, OpenSourceUtil (com.intellij.util)
navigate:51, OpenSourceUtil (com.intellij.util)
openSourcesFrom:19, OpenSourceUtil (com.intellij.util)
onDoubleClick:46, MyDoubleClickListener (com.jetbrains.bigdatatools.rfs.projectview)
onClick:30, DoubleClickListener (com.intellij.ui)
mouseReleased:59, ClickListener$1 (com.intellij.ui)
mouseReleased:290, AWTEventMulticaster (java.awt)
mouseReleased:289, AWTEventMulticaster (java.awt)
mouseReleased:289, AWTEventMulticaster (java.awt)
mouseReleased:289, AWTEventMulticaster (java.awt)
processMouseEvent:6539, Component (java.awt)
processMouseEvent:3324, JComponent (javax.swing)
processMouseEvent:391, Tree (com.intellij.ui.treeStructure)
processMouseEvent:43, DnDAwareTree (com.intellij.ide.dnd.aware)
processEvent:6304, Component (java.awt)
processEvent:2239, Container (java.awt)
dispatchEventImpl:4889, Component (java.awt)
dispatchEventImpl:2297, Container (java.awt)
dispatchEvent:4711, Component (java.awt)
retargetMouseEvent:4904, LightweightDispatcher (java.awt)
processMouseEvent:4535, LightweightDispatcher (java.awt)
dispatchEvent:4476, LightweightDispatcher (java.awt)
dispatchEventImpl:2283, Container (java.awt)
dispatchEventImpl:2746, Window (java.awt)
dispatchEvent:4711, Component (java.awt)
dispatchEventImpl:760, EventQueue (java.awt)
access$500:97, EventQueue (java.awt)
run:709, EventQueue$3 (java.awt)
run:703, EventQueue$3 (java.awt)
doPrivileged:-2, AccessController (java.security)
doIntersectionPrivilege:74, ProtectionDomain$JavaSecurityAccessImpl (java.security)
doIntersectionPrivilege:84, ProtectionDomain$JavaSecurityAccessImpl (java.security)
run:733, EventQueue$4 (java.awt)
run:731, EventQueue$4 (java.awt)
doPrivileged:-2, AccessController (java.security)
doIntersectionPrivilege:74, ProtectionDomain$JavaSecurityAccessImpl (java.security)
dispatchEvent:730, EventQueue (java.awt)
defaultDispatchEvent:908, IdeEventQueue (com.intellij.ide)
dispatchMouseEvent:846, IdeEventQueue (com.intellij.ide)
_dispatchEvent:778, IdeEventQueue (com.intellij.ide)
lambda$dispatchEvent$8:424, IdeEventQueue (com.intellij.ide)
computePrioritized:698, CoreProgressManager (com.intellij.openapi.progress.impl)
dispatchEvent:423, IdeEventQueue (com.intellij.ide)
pumpOneEventForFilters:205, EventDispatchThread (java.awt)
pumpEventsForFilter:116, EventDispatchThread (java.awt)
pumpEventsForHierarchy:105, EventDispatchThread (java.awt)
pumpEvents:101, EventDispatchThread (java.awt)
pumpEvents:93, EventDispatchThread (java.awt)
run:82, EventDispatchThread (java.awt)

To Reproduce I don’t have steps to reproduce, but I can give you any information you need.

Code Snippet Several potentially useful code snippets:

FileInfoAsSeekable#readFully:

  override fun readFully(position: Long, buffer: ByteArray, offset: Int, length: Int) {
    fileInfo
      .readStream(position)!!
      .use { it.read(buffer, offset, length) }
  }

where it is input stream from blob, opened as

FileInfo#readStream:

client.openInputStream(BlobRange(offset), null)

where client is BlobClient.

By specification InputStream#read should read as much as possible until buffer is full or input stream is ended. But in debugger I can see moment when offset + size > length and at this moment execution fails inside ByteBuffer#get

While readFully is being called with

offset = 0
buffer = {byte[40182088]@24505} 
length = 40182088
position = 31324

At some point it leads to ByteBuffer#get with

dst = {byte[40182088]@24505} 
offset = 37748736
length = 4194304

which is obviously out of bounds (which leads o error). It easily can be seen from stack trace that there is no eny intermediate code between my readFully method and Azure’s read method and by spec it should not lead to errors.

Setup (please complete the following information):

• OS: Linux, Mac OS X
• IDE : IntelliJ
• Version of the Library used: 12.6.1

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[rickle-msft]

Awesome! So glad to hear it! I am going to close this issue now. Please let us know if you need any more support.

[asm0dey]

Fixed for us in latest beta, thanks!