WriteToStream - Backpressure Issue?

fast-csv version: 4.3.0

I have a function that is used to write the contents of Array of Objects to a csv file with a specific header order. For small arrays I’ve not seen a problem but I have one array with over 15,000 objects and the if I execute the below the csv file only ends up with about 4,000 of them.

None of the error handling is emitted so finding the issue is problematic,

Do you think this could be a back-pressure issue, I’m not convinced but the silent failure makes me wonder?

const { writeToStream } = require('@fast-csv/format');

function writeCSV(filename, headers, data) {
try{
    console.log('Data Contains number of items: ' + data.length');
    const ws = fs.createWriteStream(filename);
    writeToStream(ws, data, { headers: headers, alwaysWriteHeaders: true })
    .on('error', (err) => {
        console.log('Error' + err);
      });
} catch (error) {
    console.log(error);  
}

The console shows there’s over 15,000 items as expected from the source but the .csv file contains just under 4,000.

[p-obrien]

Good news, this issue seems to be related to my code not allowing enough time for the stream to fully flush down to disk. I was also using VSCode to view the CSV files when testing on my laptop and it looks like it or another process such as AV caused the file to not fully flush out every now and then.

After altering my code structure the lambda completes successfully, apologies for the run around.

[p-obrien]

My Lambda has a timeout of 15 minutes and the workstation that I can reproduce on has no timeout. Will try to step through with breakpoints etc. and see if I can find anything.