buffer size for multipart s3 downloads

I noticed recently that for a large download, the awscli (aws s3 cp s3://...) was faster than using boto3.s3.transfer.MultipartDownloader.

After running a few tests of downloading an 8GB file, it looks like maybe the size of the I/O buffer here may have something to do with it. I don’t understand why, but making that buffer size larger (e.g., 256KB or 1024KB instead of the current 16KB) seems to improve download speeds consistently for me.

Perhaps that buffer size should be increased, or maybe just made configurable? I don’t understand the pros and cons other than that making it larger seems to help for my use case.

Times for downloading an 8GB file from S3 to a g2.2xlarge instance (I just changed the number in the line of code mentioned above):

• 100 seconds with 1024KB buffer
• 106 seconds with 256KB buffer
• 118 seconds with 16KB buffer (current boto3 code)
• 256 seconds with 4KB buffer

Code for testing:

import time
import boto3
import logging
from concurrent.futures import ProcessPoolExecutor

t0 = time.time()

logging.basicConfig(level='DEBUG')
logging.getLogger('botocore').setLevel('INFO')
client = boto3.client('s3')

config = boto3.s3.transfer.TransferConfig(
    multipart_threshold=64 * 1024 * 1024,
    max_concurrency=10,
    num_download_attempts=10,
    multipart_chunksize=16 * 1024 * 1024,
    max_io_queue=10000
)

config = boto3.s3.transfer.TransferConfig()
transfer = boto3.s3.transfer.MultipartDownloader(client, config, boto3.s3.transfer.OSUtils())
transfer.download_file('bucket-name', 'path/to/big/file/foo.npy', 'foo2.npy', 8000000000, {})
print("TIME: {} SECONDS".format(time.time() - t0))

I previously mentioned this here.

[kyleknap]

With the release of 1.4.0 of boto3, you now have the option to both io_chunksize and max_io_queue so for the environment where the network speed is much faster than the io speed you can configure it in a way to make io stop being the bottleneck: https://boto3.readthedocs.io/en/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig

It is important to note that with the current configuration, the defaults should be suitable. Now the io_chunksize is 256KB, which seems to be a good default value as I have found in my testing and testing from others, @gisjedi. For me with the current default configurations, boto3 achieves the same speed for downloads as the CLI for large downloads on larger instances.

Closing out issue as the defaults should now be resulting in better performance and the necessary configuration parameters related to io are now exposed to tweak to make the download faster if the results from using the defaults are still not as desired.

[jamesls]

Also relevant: https://github.com/boto/boto3/pull/737