add custom uncompressed size header to `pyodide.asm.wasm` to track loading

Problem

There is yet no official way to track pyodide loading but as we discussed how to do this in #2927 it turned out that we need to know in advance the uncompressed size of pyodide.asm.wasm.

The solution discussed relies on creating a Stream and comparing processed chunks with the contentLength. But the ContentLength header only grabs the compressed file size, and in the end the processed chunks add up to more than that.

We can use magic numbers to accomodate this, but:

• it’s ugly
• the uncompressed file size can change in future release
• there is also the possibility depending on the way the server is configured that not every user gets the same compression, some may get a brotli compressed file, others with uncompatible browsers may get a gzip file etc. This messes-up with the magic number solution.

Proposal

The clean way to go would be to have a custom header on the file allowing to programmatically know pyodide.asm.wasm uncompressed size.

We believe that any feedback we can give users on the loading status of pyodide is valuable.

Would this be possible ?

[rth]

OK, it’s indeed more useful with slow download speeds where the overall time would be dominated by fetching files and two large files pyodide.asm.wasm and pyodide.asm.data (or roughly the same compressed size) are being downloaded in parallel.

So if we wanted to implement a custom header one would need to pass it to the aws s3 command here (and have some python or bash script calculating it)

I’m not sure it’s really worth it though. For a given content-encoding (e.g. gzip) the compression ratio for .wasm would be rather similar I think. For instance, in 0.21 there is a x2.9 gzip compression ratio, and it was x3.3 in 0.18 -> that’s a 13% variation. So you could just fix a x3 ratio and use that. I would argue this difference doesn’t really matter because this progress bar is not going to be very accurate anyway, and also since it’s UI, people will likely not mind if it’s off by a bit.

[fcrozatier]

I agree that fetching the file is only one part of the process, but it is still a big part and we can track the progress at this stage. I don’t know how to track the progress of the compile step but we can still await it.

In practice here is what I have when throttling down to 4G. The progress bar is tracking the fetch:

https://user-images.githubusercontent.com/48696601/183383876-2a0f1e1d-852e-4928-96eb-f29d2b438ade.mov

See the user experience on the left ? I believe what’s not helpful is to have no feedback at all.