How to Use botocore.response.StreamingBody as stdin PIPE

I want to pipe large video files from AWS S3 into Popen’s stdin. This code runs as an AWS Lambda function, so these files won’t fit in memory or on the local file system. Also, I don’t want to copy these huge files anywhere, I just want to stream the input, process on the fly, and stream the output. I’ve already got the processing and streaming output bits working. The problem is how to obtain an input stream as a Popen pipe.

I can access a file in an S3 bucket:

import boto3
s3 = boto3.resource('s3')
response = s3.Object(bucket_name=bucket, key=key).get()
body = response['Body']  

body is a botocore.response.StreamingBody. I intend to use body something like this:

from subprocess import Popen, PIPE
Popen(cmd, stdin=PIPE, stdout=PIPE).communicate(input=body)[0]

But of course body needs to be converted into a file-like object. The question is how?

[royrusso]

Not sure if this helps, but I struggled with a similar issue streaming from boto3:s3 to flask output stream. Some sample code here, may help you:

s3_response = s3_client.get_object(Bucket=BUCKET, Key=FILENAME)

def generate(result):
   for chunk in iter(lambda: result['Body'].read(self.CHUNK_SIZE), b''):
      yield chunk

return Response(generate(s3_response), mimetype='application/zip', headers={'Content-Disposition': 'attachment;filename=' + FILENAME})
`

[mslinn]

I wish it were so simple. I must have dumped at least 30 hours into this problem, and I’ve tried a variety of approaches. I’ve tried the AWS & Python communities on StackOverflow but got no response. I think this issue requires too much setup for a generic Python programmer to address.

I’m sure this will be an often-repeated question, I’m just the first to hit it. I think someone from AWS should take it up.

For my part, I’ve got a broken product in production as a result of this issue. I am willing to pay cash money for a fix.