Pausing a stream

Pausing a stream() doesn’t seem to have an effect - The parser proceeds reading the CSV as usual.

Take for example the following code:

const rs = fs.createReadStream(this.csvPath);
rs.pause();

let count = 0;
csv()
.fromStream(rs)
.on("json", (json) => {
  count++;
  console.log(count);
})
.on("done", () => {
  cb(null, count);
})
.on("error", (err) => {
  cb(err);
})

count is logged 200 times (equal to the amount of rows in the CSV) - I was expecting it not to log anything since the stream is paused before passing it over to fromStream()

[Keyang]

😄 As mentioned before, this is not correct way of doing things.

Node.js has encapsulated this in its stream implementation and we should not need to do this.

Implement your own Writable implementation is the correct way of doing this. see https://nodejs.org/api/stream.html#stream_simplified_construction

here is an example of what you want to achieve:

var tmpArr=[];
rs.pipe(csv({},{objectMode:true})).pipe(new Writable({
  write: function(json, encoding,callback){
    tmpArr.push(json);
    if (tmpArr.length===10000){
      myDb.save(tmpArr,function(){
        tmpArr=[];
        callback();
      })
    }else{
      callback();
    }
  } ,
  objectMode:true
}))
.on('finish',function(){
  if (tmpArr.length>0){
    myDb.save(tmpArr,function(){
      tmpArr=[];
    })
  }
})

~Keyang

On 23 January 2017 at 14:41, Nicholas Kyriakides [e-mail] wrote:

Got it - thanks.

Are there any plans to make “pausing” a bit more easy? I’ve got it working with your suggestions, but it feels a tad hackish.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Keyang/node-csvtojson/issues/135#issuecomment-274504943, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCs93cHFZCjsymuKhGc-qvVE3Dgh5zmks5rVLwagaJpZM4LqJLl .

[nicholaswmin]

Nice - that looks much cleaner. Thanks again.