File Size Over 30mb is crashing app

Support plan

• which support plan is this issue covered by? (e.g. Community, Sponsor, or Enterprise): community
• is this issue currently blocking your project? (yes/no):no
• is this issue affecting a production system? (yes/no):yes

Context

• node version: v10.17.0
• module (formidable) version:1.2.2 and 1.2.1
• environment (e.g. node, browser, native, OS):browser
• used with (i.e. popular names of modules):foalts, react
• any other relevant information:

What are you trying to achieve or the steps to reproduce?

Hi, Just trying to upload a file. Someone noticed an issue when they tried to upload a 65mb file but I am noticing that when I go over 30mb I get errors in the console of VS code. Please see the stack trace from the error. I was on version 1.2.1 but its not working on 1.2.2 either. If you need anymore info, I will be happy to provide it. Thank you 😃

What was the result you got?

<--- Last few GCs --->

[15084:000001BC6302C4B0]   287751 ms: Mark-sweep 1053.9 (1070.7) -> 1053.9 (1067.2) MB, 718.5 / 0.0 ms  (average mu 
= 0.602, current mu = 0.000) last resort GC in old space requested
[15084:000001BC6302C4B0]   288447 ms: Mark-sweep 1053.9 (1067.2) -> 1053.9 (1067.2) MB, 696.1 / 0.0 ms  (average mu 
= 0.413, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 000003DE8C2DC5C1]
    1: StubFrame [pc: 000003DE8C95C882]
Security context: 0x032fc8b1e6e9 <JSObject>
    2: DoJoin(aka DoJoin) [0000032FC8B05E91] [native array.js:1] [bytecode=00000333C45FB9D1 offset=182](this=0x02f50b7826f1 <undefined>,l=0x01df193c2dc1 <JSArray[60171968]>,m=60171968,A=0x02f50b7828c9 <true>,w=0x02f50b7829f1 <String[0]: >,v=0x02f50b7829a1 <false>)
    3: Join(aka Join) [0000032FC8B05EE1] [native arra...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 00007FF6E0D47DDA v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4506
 2: 00007FF6E0D22876 node::MakeCallback+4534
 3: 00007FF6E0D231F0 node_module_register+2032
 4: 00007FF6E103B6BE v8::internal::FatalProcessOutOfMemory+846
 5: 00007FF6E103B5EF v8::internal::FatalProcessOutOfMemory+639
 6: 00007FF6E1221DE4 v8::internal::Heap::MaxHeapGrowingFactor+9620
 7: 00007FF6E12202AB v8::internal::Heap::MaxHeapGrowingFactor+2651
 8: 00007FF6E134A2B8 v8::internal::Factory::AllocateRawArray+56
 9: 00007FF6E134AC32 v8::internal::Factory::NewFixedArrayWithFiller+66
10: 00007FF6E10C82DF v8::internal::HashTable<v8::internal::NumberDictionary,v8::internal::NumberDictionaryShape>::NewInternal+63
11: 00007FF6E10C8252 v8::internal::HashTable<v8::internal::NumberDictionary,v8::internal::NumberDictionaryShape>::EnsureCapacity+226
12: 00007FF6E10C8811 v8::internal::Dictionary<v8::internal::NumberDictionary,v8::internal::NumberDictionaryShape>::Add+129
13: 00007FF6E13652D7 v8::internal::Factory::NewCallHandlerInfo+49095
14: 00007FF6E10BCF88 v8::internal::SharedFunctionInfo::SetScript+23528
15: 00007FF6E109FF33 v8::internal::JSReceiver::class_name+20595
16: 00007FF6E133B539 v8::internal::wasm::WasmCodeManager::LookupCode+15273
17: 00007FF6E1248BFC std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Umove+57468
18: 000003DE8C2DC5C1

What result did you expect?

File uploaded I was hoping for. 😃

[GrosSacASac]

Use streams to encrypt files without memory overflow. Closing as it is not an issue in Formidable

[shellac85]

I dont think its a issue with Formidable. In our code there is a section that encrypts the code and I think its running out of memory with large files:

export const encryptFile: (
  fsPath: string,
  encryptionKey: string,
) => Promise<void> = async (fsPath, encryptionKey) => {
  if (!encryptionKey) throw new Error('Customer encryption key not found');

  const fileData = await readFile(fsPath);
  const dataBase64 = fileData.toString('base64');
  const encryptedFile = CryptoJS.AES.encrypt(dataBase64, encryptionKey);
  const buffer = Buffer.from(encryptedFile.toString(), 'base64');
  writeFileSync(fsPath, buffer);
};

I was trying to achieve the same with streams but no joy yet.