question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Document that s3 upload_fileobj closes the file obj

See original GitHub issue

When I look at the documentation for functions like upload_fileobj, it’s not clear that boto3 will close the file for me.

I have a use case where I have a zip file in a BytesIO, I want to upload it and unzip it. I assumed boto wouldn’t close the file for me, but it does. That’s fine, since the code unzipping this won’t close it, so I can just swap the order.

But the documentation should clarify what the behavior is, right there where it explains that the argument is a file-like object.

e.g.

Fileobj (a file-like object) – A file-like object to upload. At a minimum, it must implement the read method, and must return bytes. This function will have .close() called on it.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
mdavis-xyzcommented, Feb 17, 2022

@tim-finnigan I don’t think this ticket should be closed.

The issue is that the documentation for boto3 does not mention an unintuitive behavior of boto3. That’s still an issue. Those other Github issues for other repositories won’t fix that doc issue for this repository.

As an end user it has never been clear to me what the difference is between botocore, s3transfer and boto3. (Perhaps that’s the doc change that’s required?) I don’t think it’s reasonable to assume that users will think to look deep into the docs of 2 dependencies of boto3 to find this behavior. But if they did, they still wouldn’t find it, because that’s not what those other issues are for.

That 80 issue in botocore has remained open for half a decade. I don’t think that’s going to change any time soon. It’s not even clear what that issue is about. (is it about changing the docs to describe the file closing behavior, or changing the behavior to not close the file?)

If I submit a PR to just add one sentence to the boto3 docs, will you accept it?

0reactions
matt-telstracommented, Nov 6, 2022

Where should I put the comments?

Because Boto3 is programmatically generated from the service definition files, I find it incredibly hard to navigate the codebase and find out where anything is, or how it works.

Read more comments on GitHub >

github_iconTop Results From Across the Web

s3.upload_fileobj gives error a bytes-like object is required
My question is inspired by a previous SO about this topic: uploading and saving DataFrames as csv files in Amazon Web ...
Read more >
S3 — Boto3 Docs 1.26.34 documentation - AWS
A low-level client representing Amazon Simple Storage Service (S3) ... Key of the object for which the multipart upload was initiated. UploadId (string)...
Read more >
Using Python - IBM Cloud Docs
Upload binary file (preferred method). The upload_fileobj method of the S3.Object class automatically runs a multi-part upload when necessary.
Read more >
Usage — Async AWS SDK for Python 0.0.0 documentation
aioboto3.resource will return a boto3 like resource object, but it will also have an ... Here are some examples of uploading and streaming...
Read more >
Resolve issues with uploading large files in Amazon S3
For large files, Amazon S3 might separate the file into multiple uploads to maximize the upload speed. The Amazon S3 console might time...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found