Allow to decode more than one object from InputStream

This is a meta issue for collecting various use-cases for designing an API that will allow to actually ‘stream’ objects from InputStream, instead of decoding a single one using Json.decodeFromStream.

Please describe your use case in detail, if this is an HTTP long-polling stream or WebSocket, first-party or third-party API, or just a very large file.

[OliverO2]

My use case is using WebSocket streaming to drive the entire user interface.

• Backend -> frontend: The backend knows or tries to predict the object tree required by the frontend and sends it. The backend also sends updates to existing objects and additional objects in response to frontend requests.
• Frontend -> backend: The frontend sends requests for additional objects as necessary (e.g. lists rendering more elements than expected, additional content made visible by scrolling). The frontend also sends updates to objects resulting from user interaction.

The current implementation uses streams of Event objects (one stream per direction) and serializes/deserializes them into/from WebSocket frames via ProtoBuf. It also uses special synchronization events for backpressure (e.g. throttling the backend when it sends updates faster than the frontend can process them).

So although this is a streaming scenario, serialization is currently dealing with single objects only (events enveloped in frames). A drawback of the current approach is that a large event might have to be split into (artificially created) smaller chunks before serialization, to allow for fine-grained backpressure (in-flight synchronization events) and stuff like progress indicators. Chunking can occur at the binary level, though.

[lion7]

I’d like to tail a JSON lines file from a potentially infinite stream. Examples of this format can be found here: https://jsonlines.org/examples/ and look like this:

["Name", "Session", "Score", "Completed"]
["Gilbert", "2013", 24, true]
["Alexa", "2013", 29, true]
["May", "2012B", 14, false]
["Deloise", "2012A", 19, true] 

Note that this format is somewhat different from the JSON spec. Instead of a really large array containing the objects, multiple ‘root’ objects are used where each object is separated by a line separator.

In practice we often use logback with https://github.com/logstash/logstash-logback-encoder for our logging, which writes each log event as a single-line JSON to some output such as a file or socket.