Buffering techniques?

There must be a better way to do this but I’m not sure yet. Feedback would be welcome. Hopefully this helps provide real-world feedback and uses cases for most, and a reference for future others.

let DeviceConnections = (redis) => (

  /* Immediately subscribe to ConnectionChange events
  and buffer any subscription data until the initial data
  can be queried.

  Once initial data is retreived we push it through the
  returned stream, and then also push whatever data has been
  buffered up to now.

  Finally the buffer is discarded and hereafter we simply
  pipe subscription data into the returned stream. */

  FRP
  .create((add, end, error) => {

    let isInitializing = true
    let queue = []
    let redisSubscriber = redis.duplicate()

    redisSubscriber
    .psubscribe(`${deviceChannelPrefix}*`)
    .then((/*, count */) => (
      redis
      .keys(`${connectionKeyPrefix}*`)
      .map(R.compose(mockConnectionChangeEvent(2), parseIDFromKey))
      .each(add)
      .tap(() => {
        queue.forEach(add)
        queue.length = 0
        isInitializing = false
      })
    ))
    .catch(error)

    RedisStream.jsonPmessage(redisSubscriber)
    .forEach((event) => {
      if (isInitializing) queue.push(event); else add(event)
    })

    /* Disposer function. */
    return () => {
      redisSubscriber.quit()
    }
  })
)

The goal is to effectively watch a data source while getting its current state. While the query I/O occurs state may change. As long as we catch these changes and concat them to the initial results we’re fine (if not a little time-delayed, but that is harmless).

You may be asking “why don’t we just use an industrial grade solution like RethinkDB”? We probably will in the future but for now we’re working with some legacy systems. Anyways I imagine this is a common enough pattern:

1. Create a stream and start consuming (buffer results)
2. Conduct some action
3. Put action results at head of stream
4. Flush stream buffer and continue streaming as per normal

[briancavalier]

Ok, this is totally untested, but seems like it’s on the right track 😃

I think about event streams as timelines, laid out in front of me on the table, i.e. static things. Then, I try to think about how I can cut them up and reassemble them into the new timeline, or timelines that I need. So, the approach “cuts” the changes timeline, splitting it into 2 segments: before initial data has arrived (i.e. before the promise has fulfilled), and after. That leaves 3 pieces: the initial data, the “before” segment, and the “after” segment. It then concatenates those 3, which takes some minor hoop jumping because it involves arrays, promises, and event streams.

import { fromPromise, from } from 'most';
import { concat, map } from 'ramda';
import { lift } from 'when';

// append :: [a] -> a -> [a]
let append = (a, x) => { a.push(x); return a; };

// collect :: Stream a -> Promise [a]
let collect = s => s.reduce(append, []);

// concatP :: Promise [a] -> Promise [a] -> Promise [a]
let concatP = lift(concat);

// getInitialChanges :: DataChangesEmitter a -> Promise [b]
let getInitialChanges = dataChangesEmitter => {
  return dataChangesEmitter.watch().then(map(asChanges));
}

// fromPromiseArray :: Promise [a] -> Stream a
let fromPromiseArray = p => fromPromise(p).chain(from);

function streamChanges(dataChangesEmitter) {
  // All the changes, sans initial
  let changes = streamify(dataChangesEmitter);

  // Get promise for initial data
  let initial = getInitialChanges(dataChangesEmitter);

  // Make a signal that fires when initial data has arrived
  let initialReady = fromPromise(initial);

  // "Split" changes into events before initial data
  // has arrived, and after
  let before = changes.until(initialReady);
  let after = changes.since(initialReady);

  // Concatenate the initial data with changes that arrived
  // before initialReady.
  let initialAndBefore = fromPromiseArray(concatP(initial, collect(before)));

  // Finally, append after
  return initialAndBefore.concat(after);
}

Let me know if that makes any sense at all, and whether it might be helpful!

[jasonkuhrt]

Hey @briancavalier it has been a while. Dusting off the code for review, this is what it became:

// Main :: { redis :: Redis, redisSubscriber :: Redis } -> [Promise [DeviceID], Stream ConnectEvent, Stream DisconnectEvent]
const Main = ({ redis, redisSubscriber }) => {

  const [
    streamRequest,
    rawChangesStream
  ] = redisSubscriber.psubscribeStream(deviceChannelPattern)

  const changesStream = rawChangesStream.map(JSON.parse)
  const willInitializeDataPromise = streamRequest.then(() => fetchCurrentConnections(redis))
  const didInitializeDataSignal = FRP.fromPromise(willInitializeDataPromise)
  const fixedChangesStream = FRP.concat(
    /* changesStream data buffered during initial data fetch. */
    FRP.fromPromise(
      changesStream
        .takeUntil(didInitializeDataSignal)
        .collect()
    ).chain(FRP.from),
    /* changesStream data _after_ initial data fetch.  */
    changesStream
      .skipUntil(didInitializeDataSignal)
  )

  return [
    willInitializeDataPromise,
    fixedChangesStream.filter(isConnect),
    fixedChangesStream.filter(isDisconnect)
  ]
}