Crash for parsing JSON in large CrossRef query (116,808)

I was trying to collect a large set of metadata and getpapers seemed to have trouble parsing the metadata. I had sufficient RAM remaining on my machine, so it might be a limitation of JSON parsing. I copied the query and error below. Not crucial for me right now (I’ll loop through the years instead of a large one), but cutting into smaller queries might become problematic when a restricted query also yields large number of results (e.g., month query in 2015).

Query

getpapers --api crossref -o cr-res --filter "type:journal-article,prefix:10.1016,from-pub-date:1000,until-pub-date:1886"

Error

info: Searching using crossref API
info: Found 116808 results
info: Saving result metadata
/usr/local/lib/node_modules/getpapers/lib/crossref.js:119
  var pretty = JSON.stringify(crossref.allresults, null, 2)
                    ^

RangeError: Invalid string length
    at join (native)
    at Object.stringify (native)
    at CrossRef.handleSearchResults (/usr/local/lib/node_modules/getpapers/lib/crossref.js:119:21)
    at pageQuery (/usr/local/lib/node_modules/getpapers/lib/crossref.js:41:16)
    at /usr/local/lib/node_modules/getpapers/node_modules/crossref/index.js:92:5
    at Request._callback (/usr/local/lib/node_modules/getpapers/node_modules/crossref/index.js:31:5)
    at Request.self.callback (/usr/local/lib/node_modules/getpapers/node_modules/request/request.js:198:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/usr/local/lib/node_modules/getpapers/node_modules/request/request.js:1082:10)

[tarrow]

I’m just testing pushing up the memory limits with node --max-old-space-size=70000

My conclusion is that is doesn’t fix the issue. This is interesting because the way I read the SO links suggest that it throws this error because we hit the limit of the heap size. I bumped my heap size to 70GB. The RSS and VSZ of the process rose to a large number but not higher than 70GB.

RSS    VSZ
1189760 2376404

Just under 24 GB.

It errors at the same point as you both found on the stringify call. I’m going to look into why this happens.

[chartgerink]

Thanks, batching is what I am doing (looping per year), but wanted to check whether this was something that needed fixing. Apparently a larger upscale problem of node (which makes sense given the size of the JSON files that are returned from crossref).