method to propagate backpressure from parser handler

First some background on this idea (coming from the image PR):

Currently SIXELs are decoded synchronously and the image decoding can correctly slow down the whole input processing chain, which leads to a drop of callback calls in WriteBuffer. Ok - sync processing works.

Ofc image decoding creates quite alot fuzz on the main thread and prolly should be done in a worker (for PNG/JPEG it has to be done async anyway). And thats where the data pressure hell begins - the SIXEL decoder has a throughput of ~50 MB/s while the DCS parser path operates at ~250 MB/s, means the terminal floods the decoder with 200 MB more data than it can process in a second (worst case). Currently there is no way to inform the WriteBuffer asynchronously about that data pressure condition from within the terminal.

Repro: Doing cat *.sixel in a folder full of sixel images with my async decoder triggers the issue. Same with mpv with their new sixel video output above a certain output size.

In general: This is currently a non-issue for our normal sync handler actions, as sync code always “blocks” and would indicate data pressure by a lower callback rate in WriteBuffer. But any async handler code with lower throughput than the parser itself will run into this. It is def. not only a problem of image processing, it just happened to be the first complicated async handler.

Idea: It would be good to have some way to pause/resume the data chunk processing on WriteBuffer from a parser handler. Given that flow control is properly set up for the terminal itself, this enables correct backpressure signalling from async parser handler code as well.

Up for discussion.

Edit: I dont need this fine-grained to a single input buffer position (although this would solve many issues around async handlers in general, the needed stack unwinding is not worth it), I think our already established chunk border stops are good enough.

[mofux]

I can’t provide much technical input as I’m not very familiar with the parser, but I enjoyed watching this issue on its journey over last couple of days. Thanks for sharing all the updates, it really helped me to get an understanding of the technical solution you are striving for. I think your latest iteration looks pretty good.

[jerch]

T | Promise<T> should be possible. And yes, async handlers should be used with great care only, they will slow down the input processing alot, no matter how we deal with the state internally in the end. This mainly results from the needed “code slicing” and re-scheduling of the continuation (+ promise creation in case of async parse methods) - the higher the async load is, the higher gets this “bookkeeping”.

Need a few more tests with the conceptual ideas before getting down to a PR.