Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Using node streams with highland

See original GitHub issue

I often need to use node-csv and JSONStream for processing large files. What is the best way to use these node-style streams with highland so that back-pressure is managed properly?

highlandStream.pipe(csvStream) returns the destination stream which isn’t a highland stream so I can’t continue chaining.

I was finally able to get it to work, but it wasn’t easy:

// Use _(source) to create a generator
function getFeatures(filename){ 

  var _push, _next;

  // Setup JSONStream that generates many events
  var featureStream = fs.createReadStream(path.join(sourceDir, filename))
        .on('data', function(feature){
          // Pause the stream until the generator is called again
          // to manage back-pressure properly
          _push(null, feature);
        .on('end', function(){
          _push(null, _.nil);

  return _(function(push, next){    
    _push = push;
    _next = next;
    // Resume the stream to get the next data event from the json stream

Besides being a little difficult to setup, it can only be used at the beginning of a stream. If I want to process multiple files this way then I have to concoct another hairy beast that enables each event to spawn a new stream and only thunk when that new stream is done.

_(filenames).consume(function(error, filename, outerPush, outerNext){
  if(filename === _.nil){
    outerPush(null, _.nil);
  } else {
      .consume(function(error, feature, innerPush, innerNext){
        if(feature === _.nil){
          innerPush(null, _.nil);
          // Push the filename out so that we can thunk
          // and get the data moving
          outerPush(null, filename);
          // Let the outer stream know we're done
        } else {
          innerPush(null, feature);
        // Need to call this to thunk and get data moving
  // Need to call this to thunk and get data moving

Is there a better way to handle these situations with the current highland api?

Either way, highland is still making our life a lot easier. Thanks for creating it.

Issue Analytics

  • State:closed
  • Created 10 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

caolancommented, Feb 18, 2014

@justincy yes, you’ll get buffers from fs.createReadStream, if you want strings then convert them to a string by calling .toString() on the buffers:

_(fs.createReadStream(filename)).invoke('toString', []).each(_.log);
justincycommented, Feb 18, 2014

Oh, right, thanks.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Node Readable Stream - Pass in a Node Readable Stream object to wrap it with the Highland API. Reading from the resulting Highland...
Read more >
caolan/highland: High-level streams library for Node ... - GitHub
The high-level streams library for Node.js and the browser. View the Highland website for more in-depth documentation. build status Join the chat at...
Read more >
pipe node streams in parallel with highland js - Stack Overflow
I have an array of async node streams that I want to consume in parallel is there a easy way to do this...
Read more >
highland.Stream JavaScript and Node.js code examples
Most used highland functions · Stream.filter. Creates a new Stream including only the values which pass a truth test. · _ · Stream.done....
Read more >
High level Node.JS streams
If you need to handle your synchronous and asynchronous data with differrent abstractions in one way, operate with Node.js streams at higher ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found