Ability to stream data (for non-serverless adapters like adapter-node)

Is your feature request related to a problem? Please describe. I have APIs built with Sapper that I’d like to migrate to SvelteKit, but some endpoints rely heavily on using Node streams to pipe data to the response, as there can be gigabytes of data that are too large to serialize to a string server-side in memory. Some of this data is coming from a database, other data is coming from streaming and processing large files that live on AWS S3.

Describe the solution you’d like I’d like to be able to return an async iterator in the body attribute of the response. I realise streaming isn’t possible with most (all?) serverless adapters or adapter-static, but would certainly be possible with adapter-node and possibly other adapters that don’t yet exist.

Describe alternatives you’ve considered adapter-node and the development server could expose the Node res object to the request so I could pipe directly to the res object inside the request handler. Otherwise, I will have to split off the API into a standalone Node server that does not live within SvelteKit, or continue to use Sapper and have a much slower build process during development.

How important is this feature to you? I would love to keep my API in SvelteKit as I currently do in Sapper. It’s really nice having the API endpoints live in the same router. To split it off into a separate server will incur extra complexity, such as dealing with sharing authentication, keeping the servers in sync, multiple code bases, etc. I would really rather not stay with Sapper long term.

Additional context Add any other context or screenshots about the feature request here.

EDITED: changed the preferred solution from stream object to async iterator, because Node Streams in v10+ are also async iterators.

[Rich-Harris]

I’ve opened #3419 which covers the use case discussed here as well as multipart forms (#70), in light of the changes in #3384. tl;dr @mrkishi convinced me that ReadableStream is preferable even if the ergonomics aren’t quite as nice, and now that we’ve switched to returning standard Response objects from handle, it makes sense for handlers to return a valid Response body, which means ReadableStream.

Closing this in favour of that issue — thanks for the valuable discussion everyone!

[benmccann]

Hi folks, I wanted everyone to know that Rich posted a mini-RFC regarding file uploads, which overlaps quite a bit with the discussion here around async iterators, etc. Please take a look