createWriteStream({resumable:false}) causes error to be swallowed and program to hang

Environment details

• OS: MacOS and Linux
• Node.js version: 10.6.0
• npm version: 6.1.0
• @google-cloud/storage version: 1.7.0

Steps to reproduce

1. Run the code below (before, change projectId and bucket in the code, and run npm install @google-cloud/storage @rauschma/stringio)
2. It should fail with error code 429 (rateLimitExceeded), but instead the code never finishes. This is the problem. The program should fail, because we’re putting the same content in the same path too many times. (If you always put the text in random paths then everything works without a 429.)
3. Comment out resumable: false and run it again
4. It will fail with error code 429, as expected.

Code:

'use strict'
const Storage = require('@google-cloud/storage')
const {StringStream} = require('@rauschma/stringio')

const projectId = 'rendering-grid'
const bucket = 'test-storage-problem-can-delete'

async function main() {
  const storage = new Storage({
    projectId,
  })

  const put = async () => {
    await new Promise((resolve, reject) => {
      const writeStream = storage
        .bucket(bucket)
        .file('foo/bar')
        .createWriteStream({
          resumable: false,
          metadata: {
            contentType: 'text/plain',
          },
        })

      writeStream.on('finish', resolve).on('error', reject)

      const readStream = new StringStream('some debugging text')

      readStream.on('error', reject)
      readStream.pipe(writeStream)
    })
  }

  for (let i = 0; i < 10; ++i) {
    console.log('#### Run #', i + 1)
    await Promise.all([...Array(10)].map(() => put().then(() => process.stdout.write('.'))))
    console.log('')
  }
}

main().catch(console.error)

So {resumable: false} is causing the program to hang, I’m guessing because it’s not reporting the error on the stream.

[stephenplusplus]

The PR didn’t change the default for all methods, only util.makeWritableStream, where we set the number of retries to 0, since retrying a POST is never possible.

[zbjornson]

Think I found it: When this library tries to retry the request because of a 429 or any other error, the stream is already consumed and there’s no data available to write to the socket. Then the socket times out expecting data.

Good job finding a way to consistently reproduce the error. I think this is the same issue plaguing everyone in #27 (which we hit on a daily basis).

For the record, the new error since the “request” module was replaced by “teeny-request”:

FetchError: network timeout at: https://www.googleapis.com/upload/storage/v1/b/zb-dev/o?uploadType=multipart&name=foo%2Fbar
    at Timeout.<anonymous> (/home/zbjornson/nodejs-storage-master/node_modules/node-fetch/lib/index.js:1338:13)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
  message: 'network timeout at: https://www.googleapis.com/upload/storage/v1/b/zb-dev/o?uploadType=multipart&name=foo%2Fbar',
  type: 'request-timeout'

The issue can be reduced to the following, bypassing a lot of the storage module’s stream complexity:

const {Storage} = require("."); // nodejs-storage
const {StringStream} = require('@rauschma/stringio')
const client = new Storage({projectId: "..."});
for (let i = 0; i < 30; i++) {
        console.log(">>");
        const body = new StringStream("abc");
        client.request({
                uri: "https://www.googleapis.com/upload/storage/v1/b/zb-dev/o",
                method: "POST",
                qs: {
                        uploadType: "multipart",
                        name: "foo/bar"
                },
                multipart: [
                        {"Content-Type": "application/json", body: '{"contentType":"text/plain"}'},
                        {"Content-Type": "text/plain", body}
                ]
        }, (err, body, res) => {
                  console.log(err, body && body.id, res && res.statusCode)
        });
}
// ~10x 200 responses, then times out

Sequence of events:

1. Full requests for all 30 attempts are sent. Body looks like this (note the STREAM BODY comment):

// socket#        timestamp characters
28 2018-10-12T10:35:05.697Z 'POST /upload/storage/v1/b/zb-dev/o?uploadType=multipart&name=foo%2Fbar HTTP/1.1\r\nUser-Agent: gcloud-node-storage/2.1.0\r\nx-goog-api-client: gl-node/8.12.0 gccl/2.1.0\r\nAuthorization: Bearer xxxxxxxxxxxxxx\r\nContent-Type: multipart/related; boundary=1737f7e9-422e-47be-b872-178edd262fe5\r\nAccept: */*\r\nAccept-Encoding: gzip,deflate\r\nConnection: close\r\nHost: www.googleapis.com\r\nTransfer-Encoding: chunked\r\n\r\n4a'
28 2018-10-12T10:35:05.697Z '\r\n'
28 2018-10-12T10:35:05.697Z '--1737f7e9-422e-47be-b872-178edd262fe5\r\nContent-Type: application/json\r\n\r\n'
28 2018-10-12T10:35:05.697Z '\r\n'
28 2018-10-12T10:35:05.697Z '1c'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '{"contentType":"text/plain"}'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '2'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '44'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '--1737f7e9-422e-47be-b872-178edd262fe5\r\nContent-Type: text/plain\r\n\r\n'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '3'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z 'abc' # <--- STREAM BODY
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '2'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '\r\n'
28 2018-10-12T10:35:05.698Z '28'
28 2018-10-12T10:35:05.699Z '\r\n'
28 2018-10-12T10:35:05.699Z '--1737f7e9-422e-47be-b872-178edd262fe5--'
28 2018-10-12T10:35:05.699Z '\r\n'
28 2018-10-12T10:35:05.699Z '0\r\n\r\n'
# ~150 ms pass
28 2018-10-12T10:35:05.844Z 'socket.close' 779 785 # 779 bytes read, 785 written

1. About 10 succeed.
2. About 20 receive a 429 response within about 100 ms.
3. Library retries. This time, the multipart body looks like this:

30 2018-10-12T10:35:07.927Z 'POST /upload/storage/v1/b/zb-dev/o?uploadType=multipart&name=foo%2Fbar HTTP/1.1\r\nUser-Agent: gcloud-node-storage/2.1.0\r\nx-goog-api-client: gl-node/8.12.0 gccl/2.1.0\r\nAuthorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxx\r\nContent-Type: multipart/related; boundary=5eeaec7c-dc01-4125-9ee8-8d767010571a\r\nAccept: */*\r\nAccept-Encoding: gzip,deflate\r\nConnection: close\r\nHost: www.googleapis.com\r\nTransfer-Encoding: chunked\r\n\r\n4a'
30 2018-10-12T10:35:07.927Z '\r\n'
30 2018-10-12T10:35:07.927Z '--5eeaec7c-dc01-4125-9ee8-8d767010571a\r\nContent-Type: application/json\r\n\r\n'
30 2018-10-12T10:35:07.927Z '\r\n'
30 2018-10-12T10:35:07.927Z '1c'
30 2018-10-12T10:35:07.927Z '\r\n'
30 2018-10-12T10:35:07.927Z '{"contentType":"text/plain"}'
30 2018-10-12T10:35:07.927Z '\r\n'
30 2018-10-12T10:35:07.927Z '2'
30 2018-10-12T10:35:07.927Z '\r\n'
30 2018-10-12T10:35:07.927Z '\r\n'
30 2018-10-12T10:35:07.927Z '\r\n'
30 2018-10-12T10:35:07.928Z '44'
30 2018-10-12T10:35:07.928Z '\r\n'
30 2018-10-12T10:35:07.928Z '--5eeaec7c-dc01-4125-9ee8-8d767010571a\r\nContent-Type: text/plain\r\n\r\n'
30 2018-10-12T10:35:07.928Z '\r\n'
# one minute passes
30 2018-10-12T10:36:07.931Z 'socket.close' 0 719 # 0 bytes read, 719 bytes written

These retries appear to happen twice after 60-second timeouts, before finally timing out entirely.

---

This seems like a fundamental flaw with using streams as a datasource when the sink might require a retry, and I have no great ideas for how to fix it. Brainstorming:

• Accept that streams can’t be retried, and remove the retry attempts for non-resumable uploads.

• Use the resumable upload mechanism and send chunks that reasonably fit in memory (ideally user-configurable) so that the chunk can be retried if needed. (For reference, Node.js’s default highWaterMark is 16,384 B.) Even with keepalive, the overhead of this might be insurmountable. A 1 GB file in 16k chunks = 61k requests.