Possible to parse uploaded files while they are being uploaded?

I’m using Starlette to write code that accepts a potentially huge CSV file upload from a user and writes that CSV file into a SQLite database: https://github.com/simonw/datasette-upload-csvs

I’m adding progress bars (via XHR) right now - https://github.com/simonw/datasette-upload-csvs/pull/5 - and the progress bar makes it obvious that after the CSV has uploaded there’s a significant pause while it is being processed.

This is because when I run await request.form() Starlette handles the entire upload, writes it to disk, then hands me a SpooledTemporaryFile I can start working with.

CSV is a format that is very supportive of processing as a stream. Is there a sensible way, working with Starlette and multipart/form-data, to stream the bytes that are being uploaded to my own handler function while the upload is continuing? Some kind of async for chunk in file_being_uploaded mechanism?

It looks like I can use async for chunk in request.stream(): for this, but presumably that means I have to roll my own parsing of the incoming multipart/form-data data. So two questions:

• Is there an elegant pattern for parsing incoming multipart/form-data file uploads as a stream in this way?
• Is this a feature that Starlette could offer as part of the Starlette API?

[tomchristie]

https://github.com/simonw/starlette/commit/d53f89d39b4d6451d82e679fffbe0bea342296fc would be a good start here yup. (Alternately, have it optionally configured on __init__?)

[adriangb]

You can asynchronously stream the request data and plug that into whatever you’d like…

I feel like this is only partially true: because you have no hook to replace Request or anything else, while you can grab the stream and use it manually this breaks the rest of the ecosystem because the behavior of interacting with that request object later on is different (e.g. Request._form is never saved).

Further, you’d have to re-implement form parsing yourself, which is a tricky thing to do (about 5% of the Starlette codebase). It feels like a lot to ask of users when maybe they just wanted to add a single function call to UploadFile.write to have a progress bar.