question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`JsonPointer` quadratic memory use: OOME on deep inputs

See original GitHub issue

Adding https://github.com/nst/JSONTestSuite in https://github.com/rallyhealth/weePickle/pull/106 uncovered that JsonPointer memory usage is quadratic over depth. -Xmx8g is insufficient to create a JsonPointer for 100k opening arrays without an OutOfMemoryError.

JsonPointer seems to be a linked list where each node contains an _asString field containing the full path to that node.

Minimal test to repro the issue: 3764ff4b1a8dc2090dec57da5dc0cc2c7f8f520b

    // such as https://github.com/nst/JSONTestSuite/blob/master/test_parsing/n_structure_100000_opening_arrays.json
    public void testDeepJsonPointer() throws Exception {
        int DEPTH = 100000;
        String INPUT = new String(new char[DEPTH]).replace("\0", "[");
        JsonParser parser = createParser(MODE_READER, INPUT);
        try {
            while (true) {
                parser.nextToken();
            }
        } catch (Exception e) {
            JsonStreamContext parsingContext = parser.getParsingContext();
            JsonPointer jsonPointer = parsingContext.pathAsPointer(); // OOME
            String pointer = jsonPointer.toString();
            String expected = new String(new char[DEPTH - 1]).replace("\0", "/0");
            assertEquals(expected, pointer);
        }
    }

heap dump filled with  instances, each containing a String of the full path

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
cowtowncodercommented, Oct 12, 2022

Finally found time to fix this; will be in 2.14.0 final.

1reaction
cowtowncodercommented, Jan 16, 2022

@pjfanning If we needed cache Caffeine is great. But in this case unfortunately caches would not really work, I think. Problem is rather that of how to remove eager construction and retaining of String representation while still allowing efficient operation for common usage.

Caches are highly problematic for this usage due to various reasons, including lack of clear ownership (JsonPointer instances should not have references back to any larger retained objects, all caches need to be owned by life-cycled entities like either JsonFactory or ObjectMapper and so on).

Read more comments on GitHub >

github_iconTop Results From Across the Web

Issues · FasterXML/jackson-core · GitHub
JsonPointer quadratic memory use: OOME on deep inputs 2.14 Issue planned (at earliest) for 2.14 performance Issue related to performance problems or ...
Read more >
Class JSONPointer - PythonHosted.org
Represents exactly one JSONPointer in compliance with IETF RFC6901. ... 'JSONPointer': The attributes of the input object are used with it's peers.
Read more >
Releases - JSON for Modern C++
Project bad_json_parsers tested how JSON parser libraries react on deeply nested inputs. It turns out that this library segfaulted at a ...
Read more >
Changelog - fuchsia Git repositories
Use unsigned indizies for array index in json pointer #2203 (t-b) ... json::from_cbor does not respect allow_exceptions = false when input is string...
Read more >
9.0 Release Notes Red Hat Enterprise Linux 9
This enables the RHEL 9 KVM hypervisor to use a number of advanced security and debugging features. One of these features is SafeStack,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found