Missing lines while stream pushing too fast

While the file been watching writing too fast, the read stream might miss some lines.

The fallowing code would reproduce the issue.

const fs = require('fs')
const exec = require('child_process').execSync;
const { Tail } = require('tail');

const bufferPath = __dirname + '/buffer.log';
const outputPath = __dirname + '/output.log';

exec(`echo "" > ${bufferPath}`);
exec(`echo "" > ${outputPath}`);

const tail = new Tail(bufferPath);
tail.on('line', line => fs.appendFileSync(outputPath, line + '\r\n'));

// write input file
let input = [];
for (let i = 0; i < 100; i++) {
  input.push(`${i}: {a:1,b:2}\r\n`)
}

input.forEach((msg) => fs.appendFile(bufferPath, msg, () => {}))

As we can see the output.log had less than 100 lines. Bigger number we set to for-loop, more lines would been missed.

This might cause Node.js streaming usage problem, we might should use pipeline instead of on('data', cb)

Backpressuring in Streams - https://nodejs.org/en/docs/guides/backpressuring-in-streams/

[JasinYip]

@nicolas-goudry You are amazing, man! I will try it later.

[lucagrulla]

Although I tested on multiple OSes I can’t replicate the issue. Closing it.