How to write a transducer?

Hi, I want to implement Blip messaging protocol (https://github.com/couchbaselabs/BLIP-Cocoa/blob/master/Docs/BLIP%20Protocol.md, it’s like TCP over WebSockets).

The data stream looks like this: -a-b-c-d-e-> and the output are messages -----M-----> One message consist of many data chunks. It is specified by the protocol when the chunks are combined and outputted as a Message.

How can I implement something like this using most? I guess it should be a transducer. I worked with nodejs Transform stream and it worked perfectly. What’s the eqivalent of https://nodejs.org/api/stream.html#stream_class_stream_transform ?

I had an attempt to write a transducer using the following code copied from transducer-js but I have no idea how to proceed further:

var most = require('most')
var mns = require('most-node-streams');

function Filter(f, xform) {
    this.xform = xform;
    this.f = f;
}

Filter.prototype['@@transducer/init'] = function () {
    return this.xform['@@transducer/init']();
};

Filter.prototype['@@transducer/result'] = function (v) {
    return this.xform['@@transducer/result'](v);
};

Filter.prototype['@@transducer/step'] = function (res, input) {
    if (this.f(input)) {
        return this.xform['@@transducer/step'](res, input);
    }
    return res;
};

var x = mns
    .fromStream(sourceFile.pipe(csvParser))
    .transduce(Filter(true /* testing... */));

most.forEach(console.log, x);

The code basically throws an error that xf is not defined. It’s not. I want something simple like NodeJS .pipe. Look:

var rawCandlesStream = sourceFile
    .pipe(csvParser)
    .pipe(new StringArrayToCandleTransformer())

[episage]

I ended up using https://www.timescale.com/ (postgresql plugin really) to bucked the tick data. Look at https://blog.timescale.com/analyzing-ethereum-bitcoin-and-1200-cryptocurrencies-using-postgresql-3958b3662e51 for examples. Much faster than processing ticks in node.

Thanks for the video link, I watched it

[briancavalier]

Hi @episage. It’s unfortunate that the word stream is quite overloaded. If I had it to do over again, I’d not use it to describe the data structure that most provides. I’m sorry for any confusion it’s caused.

Most streams and node streams have different goals and intended use cases. For example, most streams are targeted primarily at discrete events (e.g. mouse clicks), and node streams are targeted primarily at chunked data. Both can be used for other purposes, of course, but you’ll start feeling the tension as you get further from each’s intended use case. They also have different APIs: pipe() is a node stream API, and it’s not a goal of most to support it, just as until() and awaitPromises() are most APIs, and it’s unlikely such things would be commonly useful for chunked data stream

I’d recommend looking for a chunked data stream abstraction that fits your use case and works in browsers. Perhaps there is a polyfill or other userland implementation of the w3c streams proposal that would be helpful for your use cases.