Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Make large files great again

See original GitHub issue

I was in the process of stress testing this app before making the switch but I seem to have run into some problems with large files.

I don’t expect to upload this large of a file but I tried to upload a 23GB mkv file through my admin sharex config. This was directly to my publicip:9999, no nginx inbetween. Monitoring the ram and cpu, cpu seemed to spike a few times throughout the upload but ram was pretty strong at 230~MB total usage and it wouldn’t go up which is great. I watched the file gain in size in /uploads and started counting how long it would take from the upload being finished to it giving me the url. It was stuck on the process that was done uploading but still waiting to give a url for about a minute and 30 seconds. After that the file started over with reuploading, although in my uploads folder, the old file remained and a new file was being written for this new transfer.

I cancelled it then since I think it would just loop over and over probably. I tested beforehand with a gigabyte file and it worked all right (there was some delay between the upload being finished to url generated). In my config I have maxSize: '150000MB', and noJsMaxSize: '100GB', The other main options are pretty much default.

Is there a way that I could give you helpful debug logs for these transfers? I’m worried I might be hitting a bottleneck with the architecture of node.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

BobbyWibowocommented, May 28, 2020

Just a suggestion: Have you considered using a multi-writer when upload? So rather than treating uploading, and hashing, as two separate steps, it can instead be done at the same time. This is hope kipp handles file uploads.

Frankly hadn’t given that any thought before this issue. Also thought of something like that while this was going on, but I was sure Multer (the current lib we use to parse multipart data) didn’t have stream-based API to hook into. I just gave it a look again, and indeed stream-based API is only on RC versions versions atm. Can probably give that a try as-is, but eh, dunno. Though there are some solutions that involve writing own Multer storage engine, such as this one. Probably a better choice for now.

There’s a few Blake3 implementations here

As @camjac251 suggested, it might also be a good idea to use blake3. md5 should never, ever be used anymore.

Aight, that sounds good to me as well.

camjac251commented, May 29, 2020

This is incredible. Thank you for adding this. It was almost instant after the 23GB file was done uploading until it gave me the link. So much faster now

Read more comments on GitHub >

github_iconTop Results From Across the Web

What's The Best Way to Handle Extremely Large (1GB+) Text ...
Solution 1: Download a Dedicated Large File Viewer · Solution 2: Split the Large File into Smaller Chunks · Solution 3: Get a...
Read more >
How to save and share ridiculously large files - CNET
You can, with little effort, dump large files in a number of places. The usual caveat there is that there tends to be...
Read more >
The 9 Best Ways to Send Large Files - HubSpot Blog
There are plenty of free online services that make uploading and sending large files both easy and incredibly quick.
Read more >
11 Best Ways to Transfer, Share or Send Large Files in 2022
Discover the 11 Best Ways to Transfer, Send or Share large files. Pros & Cons, Free & Paid tools. Services you can start...
Read more >
Best Cloud Storage For Large Files 2022: Bigger is Beter
Whether you need to store complicated CAD drawings, upload massive media files or keep gorgeous high-resolution textures, these online storage ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found