Memory usage with large gallery exceeds maximum heap size

Bug report

Thumbsup version: 2.1.0
Node version: 6.11.4
NPM version: 3.10.10
Operating system: Debian Stretch

When the source folder is large, thumbsup uses a very large amount of memory, exausting the maximum heap space:

Finding media       [=========================] 53653/53653 done
Updating database   [===================------] 40596/53653 (10min left)
<--- Last few GCs --->

 2048680 ms: Mark-sweep 1379.3 (1434.5) -> 1379.3 (1434.5) MB, 9626.2 / 0.0 ms [allocation failure] [GC in old space requested].
 2058312 ms: Mark-sweep 1379.3 (1434.5) -> 1379.3 (1434.5) MB, 9631.8 / 0.0 ms [allocation failure] [GC in old space requested].
 2067937 ms: Mark-sweep 1379.3 (1434.5) -> 1379.2 (1410.5) MB, 9624.0 / 0.0 ms [last resort gc].
 2077559 ms: Mark-sweep 1379.2 (1410.5) -> 1379.2 (1410.5) MB, 9621.7 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xbb37e0cf781 <JS Object>
    1: exec [native regexp.js:~116] [pc=0x314498d1267] (this=0x29724e1ff289 <JS RegExp>,D=0x15ab93086591 <String[50]: ======== 2015/Prag/Kers/P1220637.JPG [40597/53653]>)
    2: /* anonymous */ [/home/www-bilder/.local/lib/node_modules/thumbsup/node_modules/exiftool-json-db/lib/exiftool.js:38] [pc=0x314498c7084] (this=0x38acf178c809 <JS Global Object>,line=0x15ab93086591 <String[50]: ======== 20...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x109f13c [node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [node]
 7: v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>) [node]
 8: v8::internal::RegExpCompiler::Assemble(v8::internal::RegExpMacroAssembler*, v8::internal::RegExpNode*, int, v8::internal::Handle<v8::internal::String>) [node]
 9: v8::internal::RegExpEngine::Compile(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::RegExpCompileData*, v8::base::Flags<v8::internal::JSRegExp::Flag, int>, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, bool) [node]
10: v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, bool) [node]
11: v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>) [node]
12: v8::internal::RegExpImpl::Exec(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::JSArray>) [node]
13: v8::internal::Runtime_RegExpExec(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x314493092a7
Aborted

I can work around that with node --max-old-space-size=4096 for now, but I would really like to be able to run thumbsup on a machine with less available RAM. When the heap size is increased, the residual memory usage hovers around 5G (metadata.json is 207M).

If this is due to metadata being kept in memory, maybe something like SQLite would be helpful?

[rprieto]

however, it does run very fast and uses little memory

Haha all good then! 😆 That’s weird it was working well for me. I’ll have another look.

Regarding RAM usage I agree it should be using a lot less! What I meant was that just solving globbing won’t be enough, and the main thumbsup resident usage sounds trickier, so I might try to focus on that one first. In the meantime good idea I’ll add a note to the README.

[rprieto]

Hi @tribut. Version 2.2.0 is now out. It should use about 1/2 the amount of RAM as version 2.1.0, and removes the high memory peaks. Changes include:

• switch to SQLite instead of JSON for the index/cache
• stream exiftool results instead of buffering them
• avoid building large arrays in memory if the data will be discarded
• fix the classic theme which had a recursive memory-bloat

Please let me know how it looks on your large gallery!