Better S3 download_fileobj docs, note potential need to call flush() with threaded transfers

Threaded transfers using the S3 download_fileobj will leave the file position in a nondeterministic state.

The example from the function’s docstring is:

import boto3
s3 = boto3.client('s3')

with open('filename', 'wb') as data:
    s3.download_fileobj('mybucket', 'mykey', data)

Inside the with clause a data.tell() call will behave differently if threaded transfers were used. This is made worse by the threshold put in place to guard against threaded transfers. For small files, the file position will always appear to be deterministic.

If the same approach is used with an open context (e.g. a named temporary file), the download could appear to be incomplete:

import boto3
import tempfile
s3 = boto3.client('s3')

with tempfile.NamedTemporaryFile(mode='wb') as data:
    s3.download_fileobj('mybucket', 'mykey', data)
    # do something with data before it's closed and removed

Noting this behavior and recommending the file is flushed prior to use would help catch downloads that appear to be incomplete.

def download_fileobj(self, Bucket, Key, Fileobj, ExtraArgs=None,
                     Callback=None, Config=None):
    """Download an object from S3 to a file-like object.

    The file-like object must be in binary mode.

    This is a managed transfer which will perform a multipart download in
    multiple threads if necessary. This behavior may leave the file position
    in an unexpected state. A call to `flush` may be required.

    Usage::

        import boto3

[nbargnesi]

We encourage you to check if this is still an issue in the latest release.

It is.

for _ in range(2):
    with open('filename', 'wb') as data:
        s3.download_fileobj(bucket, key, data)
        print(data.tell())
33554432
25165824

Using the latest boto3 version:

print(boto3.__version__)
1.21.18

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.