Proposal Stream.buffer(highWaterMark)

I propose Stream.buffer as a new method that mimics the functionality of the node stream writev function

const deleteKeys = new Writable({
	highWaterMark: 50,
	write: (key, encoding, callback) => {
		//handle 1 item in stream
	},
	writev: (keys, callback) => {
		//keys contains all keys that have been buffered since the last iteration started
	},
});

This fills a similar role to batch. the difference is that batch waits for the provided count to be pushed to it before it pushes the array downstream. buffer would collect items until downstream requests another batch up to a limit

_(source)
  .buffer(50)
  .tap(batch => {
    //batch contains all items that have flown downstream while tap was processing its last iteration (limit 50)
  });

I am willing to write this assuming it is something the project is willing to accept.

[vqvu]

Yes. That should be the case.

BTW, I suggest you use flatMap(…) instead of .map(…).mergeWithLimit(1). They are equivalent, but flatMap is clearer, and potentially more efficient.

On Thu, Oct 18, 2018 at 9:32 AM Stephen Dahl [e-mail] wrote:

I can go with batchUpTo.

This whole thing depends on my correct understanding that for this pipeline

_(source) .batchUpTo(50) .tap(batch => _(somethingAsync)) .mergeWithLimit(1)

tap would only request the next value from batchUpTo once mergeWithLimit finishes with the previous substream.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/caolan/highland/issues/655#issuecomment-431032092, or mute the thread https://github.com/notifications/unsubscribe-auth/AGIyZWv-nLNzKBM0Qm-j9xX8wIUJVuZWks5umJDqgaJpZM4XpLMH .

[vqvu]

I’m not sure why you see an infinite loop, but I left some comments in your commit.

The other problem is that you can’t add arguments to transforms. They get exported as a top-level transform that is curried, and adding additional arguments breaks that API. That is, you can run code like _.batchWithTimeOrCount(1000)(2)(stream).

I think merging it into batchWithTimeOrCount make sense, but we need to come up with another name. Maybe batchWithTimeOrCountRange? Not great, I know, but this is how things are.