question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Streaming backpressure

See original GitHub issue

Issue Description

The streaming API seems to lack support for stream backpressure. Without backpressure, it’s not possible to ensure that we’re not pushing data to the stream too quickly for it to handle. If we push data too quickly and the fflate stream can’t keep up, then it’s possible that the data will be sitting around in memory for longer than we want.

Stream backpressure helps us to detect this condition and to stop calling push() until the backed up chunks have been processed.

The example toNativeStream code used on the website doesn’t support backpressure. Is it possible to add support for this?

Or, at least expose the necessary information (i.e. outstanding chunk count, and a way to call a user-provided callback when the outstanding chunk count falls below a threshold so we can start pushing again)?

Separately, it might be helpful to just offer a WHATWG TransformStream interface to the user so they don’t need to implement it themselves.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
ferosscommented, Mar 10, 2021

Thanks! I think that counting the number of ondata callbacks will work perfectly for implementing backpressure.

0reactions
101arrowzcommented, Mar 22, 2021

Yep, what you wrote looks about right for handling backpressure on the way out. You bring up a good point, it’s not simple to use advanced features of streams with fflate, and you are correct that the catch-22 is that I can’t commit to supporting just Node.js streams or just ReadableStream because there are so many more possible ways you may want to use the output of fflate than just those two.

I’ll probably add a few more wiki entries regarding advanced usage, including backpressure support. I’ll include your snippet, since it seems to work well from my tests. I may also create another package with this code (if you’re alright with that) so it can be used more seamlessly. Thanks for your efforts!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Backpressure explained — the resisted flow of data through ...
Backpressure is when the progress of turning that input to output is resisted in some way. In most cases that resistance is computational...
Read more >
Backpressuring in Streams - Node.js
There is a general problem that occurs during data handling called backpressure and describes a buildup of data behind a buffer during data...
Read more >
What is backpressure in the context of data streaming? - Quora
It is called streaming because the material is not being saved on the device. It is drawn from a source, like Netflix and...
Read more >
Backpressure Mechanism in Spring WebFlux - Baeldung
In Reactive Streams, backpressure also defines how to regulate the transmission of stream elements. In other words, control how many elements ...
Read more >
Data Flow - Backpressure | Stream | Datacadamia - Data and Co
When the dataflow runs through asynchronous steps, each step may perform different things with different speed. In this setting, Back-pressure means a fast ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found