flatMap and flow-based programming in Highland

In this contrived example, I loop the output stream of the input back into itself creating a recursion-like behavior. It produces 1, 2, 4, 8, 16, 32, 64, 128, 256, 512.

_ = require('highland')
input = _([1])
input
  .doto((v)-> console.log(v))
  .map((v)-> v * 2)
  .filter((v)-> v <= 512)
  .pipe(input)

This doesn’t seem to work if I use a flatMap. It will only print out the first value, 1. My goal here is that I would like to return multiple values for each input value.

_ = require('highland')
input = _([1])
input
  .doto((v)-> console.log(v))
  .flatMap((v)-> return H([v * 2]))
  .filter((v)-> v <= 512)
  .pipe(input)

Is this a bug or is there something I need to do differently when using a flatMap?

[vqvu]

I thought about it some more, and there’s a way to do recursion while still maintaining the returnPipe approach.

It doesn’t look like a flow network but accomplishes the same thing.

// Applies a transform to the stream if it is not empty.
// The reverse of `otherwise`.
function ifNotEmpty(transform) {
    return function (stream) {
        return _(function (push, next) {
            stream.pull(function (err, x) {
                if (err) {
                    push(err);
                    next();
                } else if (x === _.nil) {
                    push(null, x);
                } else {
                    var nextStream = _([x])
                        .concat(stream)
                        .through(transform);
                    next(nextStream);
                }
            });
        });
    }
}

// Recursion transform.
// transform - The main transform.
// computeReturnPipe - A transform that computes what data to feed back
//     into the recursive system. It is provided with the result of the
//     main transform.
function recurse(transform, computeReturnPipe) {
    return function (stream) {
        return stream.through(recurse2)
            .flatten();
    };

    function recurse2(stream) {
        var output = transform(stream);
        var moreData = output.observe()
            .through(computeReturnPipe)
            .through(ifNotEmpty(recurse2));

        return _([output, moreData]);
    }
}

function readDir(stream) {
    return stream
        .flatMap(function (parentPath) {
            return _.wrapCallback(fs.readdir)(parentPath)
                .sequence()
                .map(function (path) {
                    return fsPath.join(parentPath, path);
                });
        });
}

function filterDir(stream) {
    return stream
        .flatFilter(function (path) {
            return _.wrapCallback(fs.stat)(path)
                .map(function (v) {
                    return v.isDirectory();
                });
        });
}

_(['/home/vqvu/Desktop'])
    .through(recurse(readDir, filterDir))
    .each(_.log)
    .done(function () {
        console.log('done');
    });

Edit: process -> transform.

[vqvu]

In the comment under “Recursion transform” you call the first argument transform but in the code it is called process

Oops! Fixed.