Streaming backpressure

The streaming API seems to lack support for stream backpressure. Without backpressure, it’s not possible to ensure that we’re not pushing data to the stream too quickly for it to handle. If we push data too quickly and the fflate stream can’t keep up, then it’s possible that the data will be sitting around in memory for longer than we want.

Stream backpressure helps us to detect this condition and to stop calling push() until the backed up chunks have been processed.

The example toNativeStream code used on the website doesn’t support backpressure. Is it possible to add support for this?

Or, at least expose the necessary information (i.e. outstanding chunk count, and a way to call a user-provided callback when the outstanding chunk count falls below a threshold so we can start pushing again)?

Separately, it might be helpful to just offer a WHATWG TransformStream interface to the user so they don’t need to implement it themselves.

[feross]

Thanks! I think that counting the number of ondata callbacks will work perfectly for implementing backpressure.

[101arrowz]

Yep, what you wrote looks about right for handling backpressure on the way out. You bring up a good point, it’s not simple to use advanced features of streams with fflate, and you are correct that the catch-22 is that I can’t commit to supporting just Node.js streams or just ReadableStream because there are so many more possible ways you may want to use the output of fflate than just those two.

I’ll probably add a few more wiki entries regarding advanced usage, including backpressure support. I’ll include your snippet, since it seems to work well from my tests. I may also create another package with this code (if you’re alright with that) so it can be used more seamlessly. Thanks for your efforts!