S3Transfer only supports files on disk, not Python file objects
See original GitHub issueThere seems to be nothing to prevent S3Transfer to upload any object that supports read, seek, tell and size, so I would like to suggest an abstraction that receives a file object. e.g. the existing code would defer to new one that is more generic
def upload_file(self, filename, bucket, key,
callback=None, extra_args=None):
self.upload_file_object(open(filename, 'r'), bucket, key,
callback=callback, extra_args=extra_args)
Issue Analytics
- State:
- Created 8 years ago
- Comments:21 (5 by maintainers)
Top Results From Across the Web
S3Transfer only supports files on disk, not Python file objects
S3Transfer only supports files on disk, not Python file objects ... There seems to be nothing to prevent S3Transfer to upload any object...
Read more >Only can upload 1 file to s3 bucket but not other files using ...
Only can upload 1 file to s3 bucket but not other files using python ; for folder in ; for files in ;...
Read more >Downloading files — Boto3 Docs 1.26.32 documentation - AWS
The download_fileobj method accepts a writeable file-like object. The file object must be opened in binary mode, not text mode. s3 = boto3....
Read more >Amazon Simple Storage Service (S3) - AWS
Amazon S3 is object storage built to store and retrieve any amount of data from ... There are certain restrictions on which buckets...
Read more >10 things you should know about using AWS S3 - Sumo Logic
If your servers are in a major data center but not in Amazon EC2, ... Alternatively, you can use S3 Transfer Acceleration to...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
+1 for me on this feature, it’s annoying have to write stream to disk when ideally I’d like to pass it straight thru to S3
The ability to upload files from file-like objects is now supported in boto3 1.4.0, I would recommend reading: https://boto3.readthedocs.io/en/latest/guide/s3.html#uploads and using one of the
upload_fileobj()
methods to upload file-like objects.I also noticed that there were discussions of implementing OSUtils so it would be good to take a look at the upgrading notes: https://boto3.readthedocs.io/en/latest/guide/upgrading.html before upgrading as functionality may have changed based on assumptions made in custom implementations of OSUtils.
Otherwise, resolving issue as the functionality is now available.