File upload randomly crashes

The bug report won’t be as thorough as I hoped and since this happens so rarely and without any notice we haven’t been able to reproduce this locally - sorry about that.

We have a service that runs a job every hour. It reads a DB and creates CSV files, and uploads them to Google Cloud Storage. Each job saves a few thousand files at about 150 KB each. A few times a month or so it randomly crashes and in our logs we find two different errors. These does not occur at the same time so might not be related, but both of them happens “randomly” and both errors include an unhandled error event in Duplexify.

Missing a file and retrying is fine, but the crash takes down our pod and makes us loose a lot of files.

The unhandled error starts in file.js in createWriteStream() when calling fileWriteStream.destroy(error).

I might be wrong there, but both of these would normally fail gracefully, but the unhandled error in Duplexify makes them crash? Would fixing this be a solution, or should one look further into the reason for destroying at the first place? Has anyone experienced similar behaviour?

Errors

events.js:292
 throw er; // Unhandled 'error' event
 ^

RequestError: The uploaded data did not match the data from the server. As a precaution, the file has been deleted. To be sure the content is the same, you should try uploading the file again. at ... /node_modules/@google-cloud/storage/build/src/file.js:1336:35 at ... /node_modules/@google-cloud/common/build/src/util.js:367:25 at Util.handleResp (... /node_modules/@google-cloud/common/build/src/util.js:144:9) at ... /node_modules/@google-cloud/common/build/src/util.js:432:22 at onResponse (... /node_modules/retry-request/index.js:214:7) at ... /node_modules/teeny-request/build/src/index.js:219:17 at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:97:5)
Emitted 'error' event on Pumpify instance at:
 at errorOrDestroy (internal/streams/destroy.js:108:12)
 at Pumpify.onerror (_stream_readable.js:753:7)
 at Pumpify.emit (events.js:315:20)
 at Pumpify.Duplexify._destroy (... /node_modules/pumpify/node_modules/duplexify/index.js:195:15)
 at ... /node_modules/pumpify/node_modules/duplexify/index.js:185:10
 at processTicksAndRejections (internal/process/task_queues.js:79:11) {
 code: 'FILE_NO_UPLOAD',
 errors: [ null ]
}

events.js:292
 throw er; // Unhandled 'error' event
 ^

Error: Upload failed at Upload.<anonymous> (... /node_modules/gcs-resumable-upload/build/src/index.js:188:30) at Upload.emit (events.js:327:22) at Upload.onResponse (... /node_modules/gcs-resumable-upload/build/src/index.js:383:14) at Upload.makeRequestStream (... /node_modules/gcs-resumable-upload/build/src/index.js:332:14) at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:97:5) at async Upload.startUploading (... /node_modules/gcs-resumable-upload/build/src/index.js:216:13)
Emitted 'error' event on Pumpify instance at:
 at errorOrDestroy (internal/streams/destroy.js:108:12)
 at Pumpify.onerror (_stream_readable.js:753:7)
 at Pumpify.emit (events.js:315:20)
 at Pumpify.Duplexify._destroy (... /node_modules/pumpify/node_modules/duplexify/index.js:195:15)
 at ... /node_modules/pumpify/node_modules/duplexify/index.js:185:10
 at processTicksAndRejections (internal/process/task_queues.js:79:11)

Code:

  const filePath = "TEST/device/system/2021/01/01/09.csv"

  storeNewFile = async (
    file: File,
    data: Data[]
  ): Promise<void> => {
    const storage = new Storage( xxx );
    const bucket = storage.bucket("myBucket");
    const file = bucket.file(filePath);
    
    const csv : string = await convertJsonToCsv(JSON.stringify(data), true);

    return new Promise<void>((resolve, reject) => {
      const input = new Readable();
      input.pipe(file.createWriteStream());

      input.on("end", resolve);
      input.on("error", reject);

      input.push(csv);
      input.push(null);
    });
  };

Environment details

• OS: node:12-alpine docker image (our prod environment)
• Node.js version: 12.22.6
• npm version: 6.14.15
• @google-cloud/storage version: 4.7.0
• Typescript: 3.7.3

EDIT: I posted wrong code snippet. We do not append to any existing files, but create a new ones.

[ddelgrosso1]

@ErlendFax using save should retry in v5 of the library. I am taking a closer look at createWriteStream as well (which should also retry).

[ddelgrosso1]

Thank you for the information @ErlendFax. I will run some experiments on our side and take a look at upload times pre/post changes. I may ultimately close out this issue and open another as it appears the original problem has been resolved.