Document that s3 upload_fileobj closes the file obj
See original GitHub issueWhen I look at the documentation for functions like upload_fileobj
, it’s not clear that boto3 will close the file for me.
I have a use case where I have a zip file in a BytesIO, I want to upload it and unzip it. I assumed boto wouldn’t close the file for me, but it does. That’s fine, since the code unzipping this won’t close it, so I can just swap the order.
But the documentation should clarify what the behavior is, right there where it explains that the argument is a file-like object.
e.g.
Fileobj (a file-like object) – A file-like object to upload. At a minimum, it must implement the read method, and must return bytes. This function will have
.close()
called on it.
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (3 by maintainers)
Top Results From Across the Web
s3.upload_fileobj gives error a bytes-like object is required
My question is inspired by a previous SO about this topic: uploading and saving DataFrames as csv files in Amazon Web ...
Read more >S3 — Boto3 Docs 1.26.34 documentation - AWS
A low-level client representing Amazon Simple Storage Service (S3) ... Key of the object for which the multipart upload was initiated. UploadId (string)...
Read more >Using Python - IBM Cloud Docs
Upload binary file (preferred method). The upload_fileobj method of the S3.Object class automatically runs a multi-part upload when necessary.
Read more >Usage — Async AWS SDK for Python 0.0.0 documentation
aioboto3.resource will return a boto3 like resource object, but it will also have an ... Here are some examples of uploading and streaming...
Read more >Resolve issues with uploading large files in Amazon S3
For large files, Amazon S3 might separate the file into multiple uploads to maximize the upload speed. The Amazon S3 console might time...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@tim-finnigan I don’t think this ticket should be closed.
The issue is that the documentation for boto3 does not mention an unintuitive behavior of boto3. That’s still an issue. Those other Github issues for other repositories won’t fix that doc issue for this repository.
As an end user it has never been clear to me what the difference is between botocore, s3transfer and boto3. (Perhaps that’s the doc change that’s required?) I don’t think it’s reasonable to assume that users will think to look deep into the docs of 2 dependencies of boto3 to find this behavior. But if they did, they still wouldn’t find it, because that’s not what those other issues are for.
That 80 issue in botocore has remained open for half a decade. I don’t think that’s going to change any time soon. It’s not even clear what that issue is about. (is it about changing the docs to describe the file closing behavior, or changing the behavior to not close the file?)
If I submit a PR to just add one sentence to the boto3 docs, will you accept it?
Where should I put the comments?
Because Boto3 is programmatically generated from the service definition files, I find it incredibly hard to navigate the codebase and find out where anything is, or how it works.