question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

S3Transfer only supports files on disk, not Python file objects

See original GitHub issue

There seems to be nothing to prevent S3Transfer to upload any object that supports read, seek, tell and size, so I would like to suggest an abstraction that receives a file object. e.g. the existing code would defer to new one that is more generic

def upload_file(self, filename, bucket, key,
                    callback=None, extra_args=None):
    self.upload_file_object(open(filename, 'r'), bucket, key,
                    callback=callback, extra_args=extra_args)

Issue Analytics

  • State:closed
  • Created 8 years ago
  • Comments:21 (5 by maintainers)

github_iconTop GitHub Comments

9reactions
hunt3ricommented, Apr 21, 2016

+1 for me on this feature, it’s annoying have to write stream to disk when ideally I’d like to pass it straight thru to S3

1reaction
kyleknapcommented, Aug 3, 2016

The ability to upload files from file-like objects is now supported in boto3 1.4.0, I would recommend reading: https://boto3.readthedocs.io/en/latest/guide/s3.html#uploads and using one of the upload_fileobj() methods to upload file-like objects.

I also noticed that there were discussions of implementing OSUtils so it would be good to take a look at the upgrading notes: https://boto3.readthedocs.io/en/latest/guide/upgrading.html before upgrading as functionality may have changed based on assumptions made in custom implementations of OSUtils.

Otherwise, resolving issue as the functionality is now available.

Read more comments on GitHub >

github_iconTop Results From Across the Web

S3Transfer only supports files on disk, not Python file objects
S3Transfer only supports files on disk, not Python file objects ... There seems to be nothing to prevent S3Transfer to upload any object...
Read more >
Only can upload 1 file to s3 bucket but not other files using ...
Only can upload 1 file to s3 bucket but not other files using python ; for folder in ; for files in ;...
Read more >
Downloading files — Boto3 Docs 1.26.32 documentation - AWS
The download_fileobj method accepts a writeable file-like object. The file object must be opened in binary mode, not text mode. s3 = boto3....
Read more >
Amazon Simple Storage Service (S3) - AWS
Amazon S3 is object storage built to store and retrieve any amount of data from ... There are certain restrictions on which buckets...
Read more >
10 things you should know about using AWS S3 - Sumo Logic
If your servers are in a major data center but not in Amazon EC2, ... Alternatively, you can use S3 Transfer Acceleration to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found