Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ujson.AsyncParser leaks memory

See original GitHub issue

Issue Description

Something’s up with AsyncParser’s memory usage. The following OOMs and dies within 10s:

import ujson.AsyncParser
import upickle.core.NoOpVisitor

object AsyncParserApp {

  def main(args: Array[String]): Unit = {
    val parser = AsyncParser[Unit]()
    parser.absorb("""{"foo":[""", NoOpVisitor)
    val chunk = """1, "omg", true, """ * 1000
    while (true) {
      parser.absorb(chunk, NoOpVisitor)
    }
  }
}

Stack trace:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at ujson.AsyncParser.resizeIfNecessary(AsyncParser.scala:111)
	at ujson.AsyncParser.absorb(AsyncParser.scala:86)
	at ujson.AsyncParser.absorb(AsyncParser.scala:96)
	at AsyncParserApp$.main(AsyncParserApp.scala:10)

Looks like it’s retaining all the input it has ever seen into the data: Array[Byte] field. Same behavior for all 3 values of AsyncParser.Mode.

Tested against 0.7.1.

Issue Analytics

State:
Created 4 years ago
Comments:6 (3 by maintainers)

Top GitHub Comments

1reaction

htmldougcommented, Nov 29, 2018

Thanks for the response and also for all your work in general.

A non-blocking parser + upickle’s Visitors + akka-streams Source/Flow/Sink is a killer combo for O(1) memory non-blocking json transformation. I’d like for the capability to continue to exist, but I don’t trust the correctness or performance of AsyncParser enough to use in prod.

Instead, I’ve been wrapping jackson’s NonBlockingJsonParser which performs faster under benchmarks and seems solid (https://gist.github.com/htmldoug/7b84b2febe67056c00199cea069003e3). I took another look at AsyncParser only to avoid porting the jackson implementation through the breaking Visitor interface changes in 0.7.1. Good interface changes, but they kept me up until 3 am last night trying to understand/replicate the usage.

Would you be interested in replacing the jawn AsyncParser with the jackson parser/visitor integration as a ujson.jackson.jvm module? It’d be significantly simpler to maintain, although couldn’t support scala.js. As a consumer, I’d love for it to exist in the same repo to keep up with future breaking API changes, particularly until the Visitor interfaces are stable as 1.x.y.

0reactions

lihaoyicommented, Dec 30, 2019

Sounds good!

On Tue, 31 Dec 2019 at 3:20 AM, Doug Roper notifications@github.com wrote:

Resolved by removing AsyncParser in 06eaccd https://github.com/lihaoyi/upickle/commit/06eaccd40bf45b3d65eddbab87dd8b7ceb354dfe. Sorry I never got around to the PR. Removal from the core makes sense to minimize.

For my needs, flowing data through OutputStream -> jackson 2.10 -> Visitor has been treating me well. It also provides debug JsonPointer info, obviating #286 https://github.com/lihaoyi/upickle/pull/286, too.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lihaoyi/upickle/issues/252?email_source=notifications&email_token=AAHEB7ECP7AM4XQPJEDTHXLQ3JCYJA5CNFSM4GG2GPFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH3ABGQ#issuecomment-569770138, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHEB7BK72GAPRVY32FYQULQ3JCYJANCNFSM4GG2GPFA .

Top Results From Across the Web

ujson.AsyncParser leaks memory · Issue #252 - GitHub

Something's up with AsyncParser's memory usage. The following OOMs and dies within 10s: import ujson.AsyncParser import upickle.core.

IT41058: MEMORY LEAK WHILE USING JSON PARSER OR ...

Memory leak observed using Accounting and Stats publishing. Memory leak was detected in JSONParserOptions object with following eyecatchers in core file ...

Memory use and speed of JSON parsers - ionel's codelog

I was using the builtin json library, and the first thing I thought - "there must be a better JSON parser". So I've...

Memory leak when using json-data on a map - Stack Overflow

Solved the problem: I had to set all Foursquare pins' title and subtitle to nil in the dealloc method, because a value (accessed...

I found a memory leak in the library! | ArduinoJson 6

There is no memory leak in ArduinoJson, you are just using it improperly. ... The “leak” appears when you reuse a JsonDocument without...