Memory exhaustion issues when deploying a large site

I’m trying to deploy a pretty large (35gb) Gatsby site for the first time to S3 via gatsby-plugin-s3 deploy with default settings and getting numerous memory exhaustion errors such as:

# Fatal error in , line 0
# API fatal error handler returned after process out of memory on the background thread

And:

class FastBuffer extends Uint8Array {}
^

RangeError: Array buffer allocation failed
    at new ArrayBuffer (<anonymous>)
    at new Uint8Array (<anonymous>)
    at new FastBuffer (internal/buffer.js:788:1)
    at createUnsafeBuffer (buffer.js:111:12)
    at allocate (buffer.js:322:10)
    at Function.allocUnsafe (buffer.js:285:10)
    at allocNewPool (internal/fs/streams.js:36:19)
    at ReadStream._read (internal/fs/streams.js:141:5)
    at ReadStream.Readable.read (_stream_readable.js:457:10)
    at ManagedUpload.fillStream (G:\dev\dunedinsound-gatsby\node_modules\aws-sdk\lib\s3\managed_upload.js:420:25)

And:

 Error: write ECONNRESET

  - stream_base_commons.js:67 WriteWrap.onWriteComplete [as oncomplete]
    internal/stream_base_commons.js:67:16

And (possibly unrelated):

 RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

  - s3.js:585 Request.extractError
    [dunedinsound-gatsby]/[aws-sdk]/lib/services/s3.js:585:35

  - sequential_executor.js:106 Request.callListeners
    [dunedinsound-gatsby]/[aws-sdk]/lib/sequential_executor.js:106:20

  - sequential_executor.js:78 Request.emit
    [dunedinsound-gatsby]/[aws-sdk]/lib/sequential_executor.js:78:10

  - request.js:683 Request.emit
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:683:14

  - request.js:22 Request.transition
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:22:10

  - state_machine.js:14 AcceptorStateMachine.runTo
    [dunedinsound-gatsby]/[aws-sdk]/lib/state_machine.js:14:12

  - state_machine.js:26
    [dunedinsound-gatsby]/[aws-sdk]/lib/state_machine.js:26:10

  - request.js:38 Request.<anonymous>
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:38:9

  - request.js:685 Request.<anonymous>
    [dunedinsound-gatsby]/[aws-sdk]/lib/request.js:685:12

  - sequential_executor.js:116 Request.callListeners
    [dunedinsound-gatsby]/[aws-sdk]/lib/sequential_executor.js:116:18

This is on Windows 10 with Node 11 64-bit and 16gb of memory.

[jariz]

@FraserThompson Good point. We should probably just pass the buffer var to ManagedUpload instead of creating a new read stream.
Can you give that a try?

Regardless, still not sure what’s causing the memory leak here, at no point are we loading the entire file into memory or something. EDIT: readFile: Asynchronously reads the entire contents of a file. Woops. Maybe we need find a better way to create the hash.

[JoshuaWalsh]

The parallelLimit function from the async library looks like a fairly nice way to limit concurrency.