Mitigating memory issues related to in memory storage

I might actually have an idea that could mitigate the problem with storing files in memory.

The problem currently is that the program needs to allocate twice the space for each file. This is because chunks comes from the browser and emits a data-event. We then store an array of all of these chunks. When this is done, the array takes up roughly as much memory as the file.

But then we do Buffer.concat(array) to create a new buffer with all of the chunks, and this creates another buffer that takes up roughly the same size as the file again.

Now, I’m not 100% sure that this is the only problem. There might be places where buffers actually gets retained longer but this is at least on problem we have.

I believe that in memory store should only be used when you expect small files, small enough to be allocated straight in the memory. (That is, all the parts in line, as a Buffer is)

What we could do is allocate one Buffer for the entire body with the size of the initial Content-Length header. Then copying in to that one buffer, and whenever we have a part, just slice that one of.

Buffer#slice returns a new buffer that points to the same actual bytes, so this would avoid allocating all files twice.

This change would require quite much reengineering and I don’t even think it’s feasible to build it upon busboy. I think that this isn’t high priority but it’s worth to keep in mind.

[LinusU]

This issue is regarding the memory storage. You can’t upload 30GB of video to the memory on the server and not expect it to take up memory.

If you are having this problem with disk storage you’ll have to open another issue.

[brown2rl]

Thanks!

-----Original Message----- From: “Cybrary” [e-mail] Sent: ‎6/‎11/‎2016 10:09 AM To: “[e-mail]” [e-mail] Subject: Python, Web App Pentesting & Social Media: Essential Resources for Every Professional

Explore Cybrary: June 11
View this email in your browser

Your Saturday Cartoon! And a Cybersecurity fun-fact.

FACT:10 percent of social media users have received a cyber-threat. More than 600,000 accounts are compromised every day on Facebook alone.

Want more? See what other info Hexis Cyber Solutions has.

Coming Soon:

Upcoming S3SS10N Wednesday: Block Cyphers vs. Stream Cyphers by Kelly Handerhan Coming June 15th Miss last week’s S3SS10N? View it here.

Web App Penetration Testing - Coming Tuesday, June 14th Cybrary’s newest course by Raymond Evans will be available in just 3 days! Stay tuned for the release - you won’t be disappointed.

Other Bits & Pieces:

Even professionals need practice. Review from our list of amazing tutorials and other resources.

Google Dorking Guide

“Done for You” - The Complete List of Kali Linux & Linux Training Videos on Cybrary

How Can I Become A H@cker?

Interested in in-depth Python tutorials? Primal Security has nearly 20 tutorials waiting to be read and watched! Check out part 1 of their series here, or find the one you want on their channel.

Copyright © 2016 Cybrary, All rights reserved. You’re receiving this message because you became a member of Cybrary IT.

Our mailing address is:

Cybrary 7833 Walker Drive, Suite 510 Greenbelt, MD 20770

Add us to your address book

Want to change how you receive these emails? You can update your preferences or unsubscribe from this list