High RAM usage on File I/O writing

I’ve been successfully using the File I/O approach in exporting Excel (.xlsx) file and then sending the file from server to the client side with res.end(buff, 'binary');

My question is : is it normal for the server to take that much RAM in writing the Excel file? If so, why does it take that much RAM? Also, according to what I experienced, the RAM usage on node.exe doesn’t go down immediately even after the res.end(buff, 'binary');

why’s that?

My file size is about 2.3MB with around 11,000 rows x 33 columns, resulting around 363,000 cells usage and the RAM node.exe used was around 600MB at the end.

It doesn’t go down unless I reset the iisnode using the iisreset command. update: node.exe goes down to normal RAM usage after around 30 mins which is still not immediately after the res.end(buff, 'binary'); update2: the memory usage also goes down if I recyle the aplication pool from IISNode I’m afraid for the fact that node.exe normally runs out of heap memory at around 1GB usage, if it doesn’t clear out its last usage and keep piling up, eventually it’ll reach that memory limit and make the node.exe process stop. How do I fix this?

here’s how I write my file to Excel

var workbook = new excel.Workbook();
var sheet = workbook.addWorksheet('Download');
sheet.columns = [
{ key: 'no', width: 30, style: {alignment: {horizontal: 'center', vertical: 'middle'}} },
{ key: 'idcovering', width: 30 },
{ key: 'kode_batch', width: 30 },
{ key: 'nokep', width: 30 },
{ key: 'kodekep', width: 30 },
{ key: 'BankAccount', width: 30 },
{ key: 'PRODUCT_CODE_PINJAMAN', width: 30 },
{ key: 'PRODUCT_CODE_PINJAMAN_DESC', width: 30 },
{ key: 'JANGKA_WAKTU_BULAN', width: 30 },
{ key: 'idcabang', width: 30 },
{ key: 'NamaCabang', width: 30 },
{ key: 'idunit', width: 30 },
{ key: 'NamaUnit', width: 30 },
{ key: 'noktp', width: 30 },
{ key: 'nopeserta', width: 30 },
{ key: 'JENIS_KELAMIN', width: 30 },
{ key: 'ALAMAT_DOMISILI', width: 30 },
{ key: 'NO_HANDPHONE', width: 30 },
{ key: 'STATUS_PENDAFTARAN', width: 30 },
{ key: 'NAMA_PENERIMA_MANFAAT', width: 30 },
{ key: 'HUB_DENGAN_PESERTA', width: 30 },
{ key: 'ALAMAT_PENERIMA_MANFAAT', width: 30 },
{ key: 'TGL_MULAI_ASURANSI', width: 30 },
{ key: 'TGL_AKHIR_ASURANSI', width: 30 },
{ key: 'JANGKA_WAKTU_COVERING', width: 30 },
{ key: 'PREMI_PER_TAHUN', width: 30 , style: { numFmt: '""#,##0;[Red]\-""#,##0' } },
{ key: 'PREMI_DIBAYARKAN', width: 30 , style: { numFmt: '""#,##0;[Red]\-""#,##0' } },
{ key: 'PREMI_BEBAN_BRI', width: 30 , style: { numFmt: '""#,##0;[Red]\-""#,##0' } },
{ key: 'PREMI_BEBAN_DEBITUR', width: 30 , style: { numFmt: '""#,##0;[Red]\-""#,##0' } },
{ key: 'NOMOR_GIRO_PREMI', width: 30 },
{ key: 'NAMA_GIRO_PREMI', width: 30 },
{ key: 'NOMOR_REKENING_DEBET', width: 30 },
{ key: 'NAMA_REKENING_DEBET', width: 30 },
]
sheet.getRow(1).font = {bold: true };
sheet.getRow(4).font = {bold: true };
sheet.getRow(5).font = {bold: true };
sheet.getRow(6).font = {bold: true };

data[0].forEach(function(item) {
sheet.addRow({
no : item.no
,idcovering : item.idcovering
,kode_batch : item.kode_batch
,nokep : item.nokep
,kodekep : item.kodekep
,BankAccount : item.BankAccount
,PRODUCT_CODE_PINJAMAN : item.PRODUCT_CODE_PINJAMAN
,PRODUCT_CODE_PINJAMAN_DESC : item.PRODUCT_CODE_PINJAMAN_DESC
,JANGKA_WAKTU_BULAN : item.JANGKA_WAKTU_BULAN
,idcabang : item.idcabang
,NamaCabang : item.NamaCabang
,idunit : item.idunit
,NamaUnit : item.NamaUnit
,noktp : item.noktp
,nopeserta : item.nopeserta
,JENIS_KELAMIN : item.JENIS_KELAMIN
,ALAMAT_DOMISILI : item.ALAMAT_DOMISILI
,NO_HANDPHONE : item.NO_HANDPHONE
,STATUS_PENDAFTARAN : item.STATUS_PENDAFTARAN
,NAMA_PENERIMA_MANFAAT : item.NAMA_PENERIMA_MANFAAT
,HUB_DENGAN_PESERTA : item.HUB_DENGAN_PESERTA
,ALAMAT_PENERIMA_MANFAAT : item.ALAMAT_PENERIMA_MANFAAT
,TGL_MULAI_ASURANSI : item.TGL_MULAI_ASURANSI
,TGL_AKHIR_ASURANSI : item.TGL_AKHIR_ASURANSI
,JANGKA_WAKTU_COVERING : item.JANGKA_WAKTU_COVERING
,PREMI_PER_TAHUN : item.PREMI_PER_TAHUN
,PREMI_DIBAYARKAN : item.PREMI_DIBAYARKAN
,PREMI_BEBAN_BRI : item.PREMI_BEBAN_BRI
,PREMI_BEBAN_DEBITUR : item.PREMI_BEBAN_DEBITUR
,NOMOR_GIRO_PREMI : item.NOMOR_GIRO_PREMI
,NAMA_GIRO_PREMI : item.NAMA_GIRO_PREMI
,NOMOR_REKENING_DEBET : item.NOMOR_REKENING_DEBET
,NAMA_REKENING_DEBET : item.NAMA_REKENING_DEBET
})
});

and to finish writing the file workbook.xlsx.writeFile("ReportDetailProduksi.xlsx").then(function() { console.log("ReportDetailProduksi file is written."); fs.readFile("ReportDetailProduksi.xlsx",function(err,buff){ fs.unlink("ReportDetailProduksi.xlsx",function(err){}); // res.setContentType("application/vnd.ms-excel"); res.setHeader('Content-disposition', 'attachment; filename=ReportDetailProduksi.xlsx'); res.end(buff); }); }).catch(function(errcatch) { console.log('errcatch', errcatch); });

[wubo02093104]

I also found the exceljs don’t release the memroy.

[mfahmirukman]

Update: I changed the way I write my xlsx file with the Streaming API provided from the package. It does consume less memory from the normal way of writing file.

Before: Memory usage with file writing : 1.3-1.5GB and the memory doesn’t get released unless I

1. Recycle the IISNode
2. Do iisreset command
3. After waiting for a while, the memory consumption goes down

After: Memory usage with the stream writing: Around 100MB-200MB at beginning. Peaking at 300MB-400MB and occasionally it goes down because of the commit()

If anyone would like to see how I do it, I can share the code.