ArrayIndexOutOfBoundsException when using Json.decodeFromStream

When using Json.decodeFromStream, the following exception occurs:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 16384 out of bounds for length 16384
	at kotlinx.serialization.json.internal.ArrayAsSequence.get(JsonLexerJvm.kt:26)
	at kotlinx.serialization.json.internal.ArrayAsSequence.charAt(JsonLexerJvm.kt:23)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:331)
	at kotlinx.serialization.json.internal.ReaderJsonLexer.consumeKeyString(JsonLexerJvm.kt:149)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:308)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeString(StreamingJsonDecoder.kt:243)
	at kotlinx.serialization.internal.StringSerializer.deserialize(Primitives.kt:142)
	at kotlinx.serialization.internal.StringSerializer.deserialize(Primitives.kt:138)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:35)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
	at eu.sweetlygeek.serialization.error.Video$$serializer.deserialize(model.kt:28)
	at eu.sweetlygeek.serialization.error.Video$$serializer.deserialize(model.kt:28)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:35)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
	at eu.sweetlygeek.serialization.error.Article$$serializer.deserialize(model.kt:7)
	at eu.sweetlygeek.serialization.error.Article$$serializer.deserialize(model.kt:7)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:35)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:535)
	at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:35)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at eu.sweetlygeek.serialization.error.ResultPage$$serializer.deserialize(model.kt:48)
	at eu.sweetlygeek.serialization.error.ResultPage$$serializer.deserialize(model.kt:48)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:35)
	at kotlinx.serialization.json.JvmStreamsKt.decodeFromStream(JvmStreams.kt:65)
	at eu.sweetlygeek.serialization.error.MainKt.main(main.kt:11)
	at eu.sweetlygeek.serialization.error.MainKt.main(main.kt)

Process finished with exit code 1

This a bug I’ve caught while using randomly-generated data in unit tests, so the data which needs to be used to reproduce is unfortunately a bit complex.

From what I’ve been able to gather (but my intuition may be false), this happens in the following code block in AbstractJsonLexer.kt:

while (char != STRING) {
            if (char == STRING_ESC) {
                usedAppend = true
                currentPosition = appendEscape(lastPosition, currentPosition)
                lastPosition = currentPosition
            } else if (++currentPosition >= source.length) {
                usedAppend = true
                // end of chunk
                appendRange(lastPosition, currentPosition)
                currentPosition = definitelyNotEof(currentPosition)
                if (currentPosition == -1)
                    fail("EOF", currentPosition)
                lastPosition = currentPosition
            }
            char = source[currentPosition]
        }

Seems like it happens when char is STRING_ESC (so we’re waiting to see what have been escaped), but at the same time we’re at the end of the chunk and nothing is loaded because we enter into the first branch of the if.

To Reproduce I’ve made a sample project with the incriminated data at https://github.com/bishiboosh/kotlin-serialization-error, the error can be replicated by launching the main method in main.kt

Expected behavior When using Json.decodeFromString, no error occurs, so decodeFromStream should not have issue with this JSON.

Environment

• Kotlin version: 1.5.31
• Library version: 1.3.0
• Kotlin platforms: JVM
• Gradle version: 7.2

[sandwwraith]

I think #1706 should fix this. You can check out the branch and test it, or I’ll check with the test case you provided a bit later

[qwwdfsad]

Fixed in 1.3.0