Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ujson.AsyncParser leaks memory

See original GitHub issue

Something’s up with AsyncParser’s memory usage. The following OOMs and dies within 10s:

import ujson.AsyncParser
import upickle.core.NoOpVisitor

object AsyncParserApp {

  def main(args: Array[String]): Unit = {
    val parser = AsyncParser[Unit]()
    parser.absorb("""{"foo":[""", NoOpVisitor)
    val chunk = """1, "omg", true, """ * 1000
    while (true) {
      parser.absorb(chunk, NoOpVisitor)

Stack trace:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at ujson.AsyncParser.resizeIfNecessary(AsyncParser.scala:111)
	at ujson.AsyncParser.absorb(AsyncParser.scala:86)
	at ujson.AsyncParser.absorb(AsyncParser.scala:96)
	at AsyncParserApp$.main(AsyncParserApp.scala:10)

Looks like it’s retaining all the input it has ever seen into the data: Array[Byte] field. Same behavior for all 3 values of AsyncParser.Mode.

Tested against 0.7.1.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

htmldougcommented, Nov 29, 2018

Thanks for the response and also for all your work in general.

A non-blocking parser + upickle’s Visitors + akka-streams Source/Flow/Sink is a killer combo for O(1) memory non-blocking json transformation. I’d like for the capability to continue to exist, but I don’t trust the correctness or performance of AsyncParser enough to use in prod.

Instead, I’ve been wrapping jackson’s NonBlockingJsonParser which performs faster under benchmarks and seems solid ( I took another look at AsyncParser only to avoid porting the jackson implementation through the breaking Visitor interface changes in 0.7.1. Good interface changes, but they kept me up until 3 am last night trying to understand/replicate the usage.

Would you be interested in replacing the jawn AsyncParser with the jackson parser/visitor integration as a ujson.jackson.jvm module? It’d be significantly simpler to maintain, although couldn’t support scala.js. As a consumer, I’d love for it to exist in the same repo to keep up with future breaking API changes, particularly until the Visitor interfaces are stable as 1.x.y.

lihaoyicommented, Dec 30, 2019

Sounds good!

On Tue, 31 Dec 2019 at 3:20 AM, Doug Roper wrote:

Resolved by removing AsyncParser in 06eaccd Sorry I never got around to the PR. Removal from the core makes sense to minimize.

For my needs, flowing data through OutputStream -> jackson 2.10 -> Visitor has been treating me well. It also provides debug JsonPointer info, obviating #286, too.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe .

Read more comments on GitHub >

github_iconTop Results From Across the Web

ujson.AsyncParser leaks memory · Issue #252 - GitHub
Something's up with AsyncParser's memory usage. The following OOMs and dies within 10s: import ujson.AsyncParser import upickle.core.
Read more >
Memory leak observed using Accounting and Stats publishing. Memory leak was detected in JSONParserOptions object with following eyecatchers in core file ...
Read more >
Memory use and speed of JSON parsers - ionel's codelog
I was using the builtin json library, and the first thing I thought - "there must be a better JSON parser". So I've...
Read more >
Memory leak when using json-data on a map - Stack Overflow
Solved the problem: I had to set all Foursquare pins' title and subtitle to nil in the dealloc method, because a value (accessed...
Read more >
I found a memory leak in the library! | ArduinoJson 6
There is no memory leak in ArduinoJson, you are just using it improperly. ... The “leak” appears when you reuse a JsonDocument without...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found