`JsonPointer` quadratic memory use: OOME on deep inputs
See original GitHub issueAdding https://github.com/nst/JSONTestSuite in https://github.com/rallyhealth/weePickle/pull/106 uncovered that JsonPointer
memory usage is quadratic over depth. -Xmx8g
is insufficient to create a JsonPointer
for 100k opening arrays without an OutOfMemoryError
.
JsonPointer
seems to be a linked list where each node contains an _asString
field containing the full path to that node.
Minimal test to repro the issue: 3764ff4b1a8dc2090dec57da5dc0cc2c7f8f520b
// such as https://github.com/nst/JSONTestSuite/blob/master/test_parsing/n_structure_100000_opening_arrays.json
public void testDeepJsonPointer() throws Exception {
int DEPTH = 100000;
String INPUT = new String(new char[DEPTH]).replace("\0", "[");
JsonParser parser = createParser(MODE_READER, INPUT);
try {
while (true) {
parser.nextToken();
}
} catch (Exception e) {
JsonStreamContext parsingContext = parser.getParsingContext();
JsonPointer jsonPointer = parsingContext.pathAsPointer(); // OOME
String pointer = jsonPointer.toString();
String expected = new String(new char[DEPTH - 1]).replace("\0", "/0");
assertEquals(expected, pointer);
}
}
Issue Analytics
- State:
- Created 2 years ago
- Comments:7 (7 by maintainers)
Top Results From Across the Web
Issues · FasterXML/jackson-core · GitHub
JsonPointer quadratic memory use: OOME on deep inputs 2.14 Issue planned (at earliest) for 2.14 performance Issue related to performance problems or ...
Read more >Class JSONPointer - PythonHosted.org
Represents exactly one JSONPointer in compliance with IETF RFC6901. ... 'JSONPointer': The attributes of the input object are used with it's peers.
Read more >Releases - JSON for Modern C++
Project bad_json_parsers tested how JSON parser libraries react on deeply nested inputs. It turns out that this library segfaulted at a ...
Read more >Changelog - fuchsia Git repositories
Use unsigned indizies for array index in json pointer #2203 (t-b) ... json::from_cbor does not respect allow_exceptions = false when input is string...
Read more >9.0 Release Notes Red Hat Enterprise Linux 9
This enables the RHEL 9 KVM hypervisor to use a number of advanced security and debugging features. One of these features is SafeStack,...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Finally found time to fix this; will be in 2.14.0 final.
@pjfanning If we needed cache Caffeine is great. But in this case unfortunately caches would not really work, I think. Problem is rather that of how to remove eager construction and retaining of String representation while still allowing efficient operation for common usage.
Caches are highly problematic for this usage due to various reasons, including lack of clear ownership (
JsonPointer
instances should not have references back to any larger retained objects, all caches need to be owned by life-cycled entities like eitherJsonFactory
orObjectMapper
and so on).