Support decoding partial streams

The following test:

  @Serializable
  class SimpleObject(val name: String)

  @Test
  fun testDecode() {
    val inputStream  = """
    {
      "name": "Object1"
    }
    {
      "name": "Object2"
    }
    """.trimIndent().byteInputStream()
    println("1")
    println(Json.decodeFromStream<SimpleObject>(inputStream).name)
    println("2")
    println(Json.decodeFromStream<SimpleObject>(inputStream).name)
  }

fails with:

Unexpected JSON token at offset 25: Expected EOF after parsing, but had { instead
JSON input: kotlinx.serialization.json.internal.ArrayAsSequence@2c4ca0f9
kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 25: Expected EOF after parsing, but had { instead
JSON input: kotlinx.serialization.json.internal.ArrayAsSequence@2c4ca0f9
	at app//kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
	at app//kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
	at app//kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:524)
	at app//kotlinx.serialization.json.internal.AbstractJsonLexer.fail$default(AbstractJsonLexer.kt:523)
	at app//kotlinx.serialization.json.internal.AbstractJsonLexer.expectEof(AbstractJsonLexer.kt:163)
	at app//kotlinx.serialization.json.JvmStreamsKt.decodeFromStream(JvmStreams.kt:66)

Is there a specific reason to mandate a trailing EOF? Being able to parse partial streams would be nice. For an example to extract a Json from a markdown file or to read an array of objects without comas like the example above.

PS: there could be a decodeFromStreamFully() that also closes the inputStream

[qwwdfsad]

Can we have a 4th option that takes a okio.BufferedSource as a parameter just like Moshi? That might even help with MPP streaming?

That’s something we’re planning to work on

[qwwdfsad]

I agree it’s fine to have the caller close the stream. My point is that if it’s the case, it feels weird enforcing a trailing EOF.

The key reason here is to InputStream being the wrong abstraction.

In order for JSON decoding to be efficient (or, at least, to be not terrible), IS has to be read in a buffered manner, otherwise, all the decoding does is round-tripping between InputStream.read() calls byte-by-byte.

As soon as one starts buffering, there is no way out – when the JSON object is read, we cannot leave the stream as is, because we could’ve read the piece of the next object that will be lost for good. IS neither provides an API “read to this buffer without consuming it + commit the offset” nor “return this N bytes back to the stream”.

Taking this into account, we basically have three options:

• Do nothing. This is the worst one, as it leaves IS completely unusable for further reads
• Close it when the decoding is done. It may or may not be convenient, e.g. because the API can be in the form “here’s the IS that has a well-formed JSON object. For the next one ask me explicitly for more and do seek”
• Just ensure that EOF is reached to let the owner of the stream decide what to do