Handling streaming request/response bodies

Describe the problem

Prior discussions: #70, #1563

As of #3384, hooks and endpoints receive a standard Request object, which among other things makes it possible to POST files:

// handles submission from a <form enctype="multipart/form-data">
export async function post({ request }) {
  const body = await request.formData();
  const file = await body.get('myfile');
  // ...
}

The file is buffered in memory, however, which means there’s a practical limit on how large it can be.

Similarly, there’s no good way to read a stream of binary data, or respond with a stream.

Describe the proposed solution

We need to solve this at a high level and at a low level. At the high level, it would be good to have utility functions for dealing with multipart form data specifically — something like this, perhaps:

import { multipart } from '$app/somewhere';

export async function post({ request }) {
  for await (const part of multipart(request)) {
    if (part.filename) {
      const uploader = create_s3_uploader(BUCKET, part.filename);
      for (const chunk of part) {
        uploader.write(chunk);
      }
    }
  }

  return { status: 201 };
}

For octet streams it might look like this:

import { stream } from '$app/somewhere';

export async function post({ request }) {
  for await (const chunk of stream(request)) {
    // ...
  }

  return { status: 201 };
}

At the low level, it should be possible to create your own stream reader and return your own ReadableStream bodies:

export async function post({ request }) {
  const reader = request.body.getReader();

  let chunk;
  while (chunk = await reader.read()) {
    if (chunk.done) break;
    await do_something_with(chunk.value);
  }

  return {
    body: new ReadableStream({
      start(controller) {
        // ...
      },
      cancel() {
        // ...
      }
    })
  };
}

Unfortunately there’s a wrinkle: ReadableStream isn’t available in older versions of Node, and the Request and Response objects (as polyfilled by node-fetch) use node streams instead of ReadableStream. It may therefore be necessary to polyfill ReadableStream during development (and in adapter-node, and lambdas) and convert between that and node streams. (On the request object, this would probably mean creating a Proxy that replaces body with a ReadableStream with the same contents as the node stream. Hopefully this is actually possible.)

Alternatives considered

No response

Importance

would make my life easier

Additional Information

No response

[0gust1]

🎉 🎉 🎉 🎉 🎉 🙏

[kaktus40]

Hello, I could stream uploading data with @sveltejs/kit version 1.0.0-next.245 from a form data.: uploader.ts is

import { readFileSync, createWriteStream } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import busboy from 'busboy';
import { pipeline } from 'stream/promises';

import type { RequestHandler } from '@sveltejs/kit';

export const post: RequestHandler = async ({ request }) => {
	const content = request.headers.get('content-type');

	const saveTo = join(tmpdir(), 'testFile');
	const bb = busboy({
		headers: {
			'content-type': content
		}
	});
	bb.on('file', (name, file, info) => {
		const { filename, encoding, mimeType } = info;
		console.log(
			`File [${name}]: filename: %j, encoding: %j, mimeType: %j`,
			filename,
			encoding,
			mimeType
		);
		file.pipe(createWriteStream(saveTo));
	});
	bb.on('field', (name, val, info) => {
		console.log(`Field [${name}]: value: %j`, val);
	});
	bb.on('close', () => {
		console.log('Done parsing form!');
	});

	await pipeline(request.body as any, bb);

	 return {
	 	status: 302,
	 	headers: {
	 		location: '/uploader'
	 	}
	 };
};

uploader.svelte is

<form method="POST" enctype="multipart/form-data" action="/authentification/uploader">
	<input type="file" name="filefield" /><br />
	<input type="text" name="textfield" /><br />
	<input type="submit" />
</form>