JsonReader: Support JSON intermixed with other content

I have a fairly large HTML page that has several JSON objects & arrays in it. I 
use a PushbackReader to read the file and handle any text I'm interested in 
(and determine when to start a JsonReader read). If I pass my Reader (or any 
reader) to JsonReader, it will read 1024 characters regardless of content. This 
means I may miss some JSON.
i.e. I may miss myObject2 in the following example because it might remain 
unprocessed in the JsonReader buffer.

<h1>some html</h1>
<script language=javascript>
var myObject1={"a":1};
alert("some non-JSON that may or may not exceed the JsonReader buffer size");
var myObject2={"b":2};

NOTE: This is a much simplified version of my problem. The JSON data is 
actually quiet large as is the HTML content.

As a temporary workaround, I've added the following method to JsonReader that 
allows me to push the unprocessed characters in the JsonReader buffer back into 
my PushbackReader (so I can start searching for "var myObject2=").

  public char[] getUnreadCharacters() {
      char[] output = new char[limit - pos];
      if (pos < limit) {
          System.arraycopy(buffer, pos, output, 0, limit - pos);
      }
      return output;
  }

I also added a constructor that allows me to modify the buffer size.

Original issue reported on code.google.com by [e-mail] on 6 Oct 2013 at 3:44

[jhugard]

Not sure if that’s a snipe or you are being serious, but that won’t work.

The issue is that gson/stream/JsonReader.fillBuffer (line 1300) reads up to 1024 bytes ahead from the caller-supplied Reader (up to buffer.length), but unconsumed bytes are never returned to that Reader. Unless I’m missing something, they are simply thrown away with the JsonReader, leaving the caller-supplied Reader in an unknown/unusable state.

https://github.com/google/gson/blob/2b15334a4981d4e0bd4f2c2d34b8978a4167f36a/gson/src/main/java/com/google/gson/stream/JsonReader.java#L1300

Supplying one byte at a time from the Reader still means up to 1024 bytes will get read from it and thrown away, leaving the Reader positioned some arbitrary number of characters past the consumed input.

Therefore, the request here is to return unconsumed characters back to the caller-supplied Reader, so that the caller has access to the next byte following the consumed JSON object (presumably using reader.mark/.reset, if reader.markSupported() is true).

[philsttr]

+1

JsonReader should not advance the underlying Reader past the end of the JSON that has been read.