Memory leak and performance issue

Generating large file (> 10000 pages) crash node on my system (Linux Fedora 23) : JavaScript heap out of memory

Here is my test script :

var PDFDocument = require('pdfkit');
var fs = require('fs');
var doc = new PDFDocument();
var v8 = require('v8');
var pageCount = 20000;

var getStats = function(text) {
  var stats = v8.getHeapSpaceStatistics();
  stats.forEach(function(stat) {
    console.log(text + ' ' + stat.space_name + ' available size : ' + stat.space_available_size);
  });
};

doc.pipe(fs.createWriteStream('test.pdf'));

console.time('Pdf generation time');
doc.on('end', function() {
  console.log();
  getStats('Final memory');
  console.log();
  console.timeEnd('Pdf generation time');
});

console.log('Start generating ' + pageCount + ' pages');
getStats('Starting memory');

for (var i = 1; i < pageCount; ++i) {
  for (var y = 0; y < 40; ++y) {
    doc.text('testText', y, y);
  }
  doc.addPage();
  if (i % 4000 === 0) {
    console.log();
    getStats('Page ' + i);
  }
}
doc.end();

And here is the result :

Start generating 20000 pages
Starting memory new_space available size : 775352
Starting memory old_space available size : 122488
Starting memory code_space available size : 231872
Starting memory map_space available size : 450104
Starting memory large_object_space available size : 1472650752

Page 4000 new_space available size : 8944344
Page 4000 old_space available size : 33672632
Page 4000 code_space available size : 1745920
Page 4000 map_space available size : 563520
Page 4000 large_object_space available size : 958758400

Page 8000 new_space available size : 9308184
Page 8000 old_space available size : 65610464
Page 8000 code_space available size : 1249696
Page 8000 map_space available size : 563432
Page 8000 large_object_space available size : 461798912

Page 12000 new_space available size : 1275960
Page 12000 old_space available size : 62591016
Page 12000 code_space available size : 782144
Page 12000 map_space available size : 564048
Page 12000 large_object_space available size : 0

<--- Last few GCs --->

[30138:0x34205a0]   218726 ms: Mark-sweep 1400.9 (1545.8) -> 1400.9 (1541.3) MB, 2200.4 / 0.1 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2201 ms) last resort 
[30138:0x34205a0]   221096 ms: Mark-sweep 1400.9 (1541.3) -> 1400.9 (1541.3) MB, 2368.5 / 0.1 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2313edd29891 <JS Object>
    2: _text [/home/dev/svn/testPdfKit/node_modules/pdfkit/js/mixins/text.js:~33] [pc=0x9b80fbc8006](this=0x3c6fdb4a5a99 <a PDFDocument with map 0x1bebc6d107f9>,text=0x19e7d0cf6569 <String[8]: testText>,x=29,y=29,options=0x38aeea002311 <undefined>,lineCallback=0x69f67fdc639 <JS BoundFunction (BoundTargetFunction 0x3c6fdb4a9e31)>)
    3: /* anonymous */ [/home/dev/svn/testPdfKit/pdfTest.js:~1] [...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x13647ec [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::SourcePositionTableBuilder::ToSourcePositionTable(v8::internal::Isolate*, v8::internal::Handle<v8::internal::AbstractCode>) [node]
 7: v8::internal::FullCodeGenerator::MakeCode(v8::internal::CompilationInfo*, unsigned long) [node]
 8: v8::internal::FullCodegenCompilationJob::ExecuteJobImpl() [node]
 9: v8::internal::CompilationJob::ExecuteJob() [node]
10: 0xd97d90 [node]
11: 0xd98b28 [node]
12: 0xd98d8f [node]
13: 0xd9ce59 [node]
14: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [node]
15: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [node]
16: 0x9b80f68437d
Abandon (core dumped)

After some analysis, i found out that the writable streams from referencePDF object eats a lot of memory. Same test result after removing the streams :

Start generating 20000 pages
Starting memory new_space available size : 2121376
Starting memory old_space available size : 600
Starting memory code_space available size : 352
Starting memory map_space available size : 450104
Starting memory large_object_space available size : 1472650752

Page 4000 new_space available size : 2915960
Page 4000 old_space available size : 21162392
Page 4000 code_space available size : 322848
Page 4000 map_space available size : 562200
Page 4000 large_object_space available size : 1162141184

Page 8000 new_space available size : 7767744
Page 8000 old_space available size : 69523024
Page 8000 code_space available size : 0
Page 8000 map_space available size : 562200
Page 8000 large_object_space available size : 877977088

Page 12000 new_space available size : 9225224
Page 12000 old_space available size : 101655856
Page 12000 code_space available size : 0
Page 12000 map_space available size : 562376
Page 12000 large_object_space available size : 589618688

Page 16000 new_space available size : 5595376
Page 16000 old_space available size : 133766616
Page 16000 code_space available size : 0
Page 16000 map_space available size : 562200
Page 16000 large_object_space available size : 301784576

Final memory new_space available size : 4856536
Final memory old_space available size : 569029752
Final memory code_space available size : 154112
Final memory map_space available size : 566336
Final memory large_object_space available size : 22240768

Pdf generation time: 82549.040ms

It seems that large_object_space available memory gets eaten up a lot more slowly and the process also goes faster : 2000 pages with streams :

Start generating 2000 pages
Starting memory new_space available size : 775320
Starting memory old_space available size : 122472
Starting memory code_space available size : 234816
Starting memory map_space available size : 450104
Starting memory large_object_space available size : 1472650752

Final memory new_space available size : 8711240
Final memory old_space available size : 13663000
Final memory code_space available size : 285504
Final memory map_space available size : 561408
Final memory large_object_space available size : 1191419392

Pdf generation time: 5894.804ms

2000 pages without streams :

Start generating 2000 pages
Starting memory new_space available size : 9768096
Starting memory old_space available size : 428104
Starting memory code_space available size : 185024
Starting memory map_space available size : 152
Starting memory large_object_space available size : 1427037696

Final memory new_space available size : 13427376
Final memory old_space available size : 14697760
Final memory code_space available size : 1355936
Final memory map_space available size : 373800
Final memory large_object_space available size : 1289543168

Pdf generation time: 4895.007ms

[rogierslag]

Any plans on merging this? The memory leak also happens when generating a large number of smaller PDF files, causing continuous service restarts

[terrisgit]

If your code generates pages faster than they can be written to the stream, or if you want to limit worst-case memory consumption, wait for pageAdded events to fire. It is of course possible to alter this example to wait after X pages have been generated. You should also catch error events and invoke r() similarly.

Unfortunately, this example runs out of memory. I will create a new ticket.

Run this program using “node --max-old-space-size=64”

const PdfDocument = require('pdfkit');

async function test() {
  const pdf = new PdfDocument({
    autoFirstPage: true,
  });

  let r; // @type {function|undefined}
  pdf.on( 'pageAdded', () => { if (r) r(); r = undefined; } );

  const output = require('fs').createWriteStream('test.pdf');
  pdf.pipe(output);

  for (let i = 0; i < 25000; i++) {
    pdf.fontSize(10).text('Some text', 100, 100);
    const p = new Promise( (resolve) => { r = resolve; } );
    pdf.addPage();
    await p;
  }

  pdf.end();
}

test().then(()=>console.log('done'), console.error);