S3.putObject only accepts streams that it can determine the length of

Is your feature request related to a problem? Please describe.

According to https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property the Body element can be a ReadableStream, however in practice, it will only succeed if the sdk can determine the length (see #2661 or https://github.com/aws/aws-sdk-js/blob/master/lib/event_listeners.js#L167). Looking at https://github.com/aws/aws-sdk-js/blob/master/lib/util.js#L198 a stream will only work if there is a path. This means that only things like fs.createReadStream will work. If the stream is transformed in any way, it will no longer work.

e.g.

Body = fs.createReadStream('./someFile').pipe(someTransform)
s3.putObject({ Bucket, Key, Body }).promise().then(console.log)

Error: Cannot determine length of [object Object]
  at Object.byteLength (aws-sdk/lib/util.js:200:26)
  at Request.SET_CONTENT_LENGTH (aws-sdk/lib/event_listeners.js:163:40)
  at Request.callListeners (aws-sdk/lib/sequential_executor.js:106:20)
  at Request.emit (aws-sdk/lib/sequential_executor.js:78:10)
  at Request.emit (aws-sdk/lib/request.js:683:14)
  at Request.transition (aws-sdk/lib/request.js:22:10)
  at AcceptorStateMachine.runTo (aws-sdk/lib/state_machine.js:14:12)
  at aws-sdk/lib/state_machine.js:26:10
  at Request.<anonymous> (aws-sdk/lib/request.js:38:9)
  at Request.<anonymous> (aws-sdk/lib/request.js:685:12)

Describe the solution you’d like

Update the documentation to more clearly identify which streams will work, and point users to S3.upload

Describe alternatives you’ve considered

A caller could include the content length, but I think that S3.upload is just a better answer.

[codeedog]

Thanks for the pointer @sPaCeMoNk3yIam. From the blog post, this is the absolute way to go for uploading from a stream. Worked like a charm. Watch your mime types, however. Also, you might have to wrap the stream if the underlying code doesn’t recognize it. I’m spooling files from a tar’d gzip’d file, and the S3 code didn’t recognize the stream as something it could use. I wrapped it in a passthrough stream.

ETA (05 JUL 2022): In case it’s not apparent from the code or my above comments, the code below uses a stream during multipart upload. The length of the file need not be known ahead of time. No complete reading of the stream - just plain old streaming.

import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
const { PassThrough } = require('stream');
const mime = require('mime-types');

//... entry is a stream

const input = {
  ACL: "public-read",  // ACL not needed if CloudFront can pull via OAI
  Bucket: bucketName,
  Key: outputPath + entry.path,
  Body: entry.pipe(new PassThrough()),
  ContentType: mime.lookup(entry.path),
}

try {
  const multipartUpload = new Upload({
      client: new S3Client({}),
      params: input,
  });

  console.log("Upload file to:", `s3://${input.Bucket}/${input.Key}`);
  await multipartUpload.done();
} catch (err) {
  console.error(err);
}

[amouly]

This is still an issue…