[storage][test] Array buffer allocation failed in Windows Node 10

Failed in in Windows Node 10 but succeeded in Node 8 & Node 12 with linux and Mac. The allocation is for a (2^31 -1) Byte buffer. And the test need to allocate about 4GB memory in all. Since it still fail when I tried to allocate a smaller buffer, I think this is due to the limited memory size of the test machine.

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=501402&view=logs&j=0421300e-d7b4-5c64-f530-fa21e83c0a48&t=be90f061-ea95-551c-fa29-249cf564c712

https://github.com/Azure/azure-sdk-for-js/blob/b1de6af33b5b2a46f6cfc2812ec0c2ebdac6d6b6/sdk/storage/storage-blob/test/node/highlevel.node.spec.ts#L346

  1) Highlevel
       uploadStream should work when blockSize = BLOCK_BLOB_MAX_STAGE_BLOCK_BYTES:
     Uncaught RangeError: Array buffer allocation failed
      at new ArrayBuffer (<anonymous>)
      at createUnsafeArrayBuffer (buffer.js:118:12)
      at createUnsafeBuffer (buffer.js:112:25)
      at allocate (buffer.js:330:12)
      at Function.allocUnsafe (buffer.js:292:10)
      at new PooledBuffer (D:\a\1\s\sdk\storage\storage-blob\dist-esm\storage-blob\src\utils\PooledBuffer.js:12:38)
      at BufferScheduler.require.BufferScheduler.shiftBufferFromUnresolvedDataArray (D:\a\1\s\sdk\storage\storage-blob\dist-esm\storage-blob\src\utils\BufferScheduler.js:221:22)
      at BufferScheduler.require.BufferScheduler.resolveData (D:\a\1\s\sdk\storage\storage-blob\dist-esm\storage-blob\src\utils\BufferScheduler.js:249:35)
      at ReadStream.<anonymous> (D:\a\1\s\sdk\storage\storage-blob\dist-esm\storage-blob\src\utils\BufferScheduler.js:160:40)
      at ReadStream.emit (events.js:198:13)
      at ReadStream.EventEmitter.emit (domain.js:448:20)
      at addChunk (_stream_readable.js:288:12)
      at readableAddChunk (_stream_readable.js:269:11)
      at ReadStream.Readable.push (_stream_readable.js:224:10)
      at lazyFs.read (internal/fs/streams.js:181:12)
      at FSReqWrap.wrapper [as oncomplete] (fs.js:467:17)

[kurtzeborn]

This is the scenario we discussed as a group and agreed it qualifies as a perf scenario which should be run through the new perf automation framework @mikeharder is standing up.

CC: @mitchdenny, @weshaggard

[mitchdenny]

You could be running into the memory ceiling on the VM. We are seeing the same thing on some on some of the Java test runs for storage. I do question the need for this test though.

What are you trying to prove/disprove by allocating an array that is so large?

Surely if you were transferring that much data down the wire as a single chunk you’d be reading it from a stream filling a much more modestly sized buffer then pushing it down the wire. Sure a customer might decide that they want to feed in a large buffer for transmission instead of streaming, but I don’t see much difference between allocating say a 100MB buffer vs. a 4GB buffer for the purposes of a unit test/live test.