Emit multiple data chunks from output Streams (was: Proper streaming)

It would be very cool if sharp could support proper streaming. As it is now, it will always emit exactly one data event which is the whole image. It kind of defeats the purposes of streams.

As I see it there is two ways to implement this.

1. The hard, but robust way.

   Every time that libvips have data that should be written, it should call a function that we have defined. We then use this data to generate a data-event on the stream. It would be really cool if libvips could provide a state-machine that has a synchronous function that gives the next chunk. That way pause and unpause would work perfectly.

2. The quick and easy fix.

   We can also create a named pipe and tell libvips to write it to that file via vips_image_write_to_file. To the user we would then return a ReadStream as returned by fs.createReadStream(path).

[gmaclennan]

@jcupitt great work on the streams branch, a shame it could never be merged. Just adding my support for any work on true streams, which would allow significantly less memory usage since right now sharp needs to load the source image into memory, and then hold the whole output buffer in memory, before writing out.

[summera]

Yes, after there’s a version of sharp that depends on the future libvips v8.9.0 we can look at the best way of handling its new Stream API.

A possible first implementation could involve accepting file descriptors as input/output, although that is likely to have to be via new API and not the standard, idiomatic Node.js Stream API (which will still be supported using the existing implementation).

A more complete solution might be to create custom libvips Stream objects and use these for passing data chunks to/from Node.js, but there are multi-threading considerations e.g. libvips will be running on a different thread to V8.

I’m planning to migrate the internals of sharp to use Node.js’s newish N-API first - see #1282 - sadly I don’t see anything in N-API for Node.js Stream objects, plus this migration will remove access to things like Node.js’ StreamBase class.

@lovell is this still on the roadmap given sharp is now using libvips 8.9?