Expose max_bandwidth configuration from s3transfer

This is a feature request to expose the new max_bandwith config option from s3transfer. It should be fairly trivial to expose the config option in boto3.s3.transfer.TransferConfig now that s3transfer supports it (as of 0.1.12)

Related issues + pull requests:

• s3transfer pull request for the max_bandwidth feature: https://github.com/boto/s3transfer/pull/101
• awscli pull request to expose config for max_bandwidth: https://github.com/aws/aws-cli/pull/2997
• awscli feature request issue for throttling s3 transfers: https://github.com/aws/aws-cli/issues/1090
• Old boto3 issue regarding throttling s3 transfers: https://github.com/boto/boto3/issues/343

[korbinianbauer]

As an alternative to tong-bluehill’s workaround, I’m using my own minimal TransferConfig class, which includes the missing max_bandwidth key. (Implementation in ▸ spoiler below, based on s3transfer.manager.TransferConfig).

This can then be passed to .upload_fileobj() or .upload_file() as per usual.

Open here for MyTransferConfig class:

from s3transfer.constants import KB, MB

class MyTransferConfig(object):
    def __init__(self,
                 multipart_threshold=8 * MB,
                 multipart_chunksize=8 * MB,
                 max_request_concurrency=10,
                 max_submission_concurrency=5,
                 max_request_queue_size=1000,
                 max_submission_queue_size=1000,
                 max_io_queue_size=1000,
                 io_chunksize=256 * KB,
                 num_download_attempts=5,
                 max_in_memory_upload_chunks=10,
                 max_in_memory_download_chunks=10,
                 max_bandwidth=None, # <--------------------------------------------------
                 use_threads=True):

        self.multipart_threshold = multipart_threshold
        self.multipart_chunksize = multipart_chunksize
        self.max_request_concurrency = max_request_concurrency
        self.max_submission_concurrency = max_submission_concurrency
        self.max_request_queue_size = max_request_queue_size
        self.max_submission_queue_size = max_submission_queue_size
        self.max_io_queue_size = max_io_queue_size
        self.io_chunksize = io_chunksize
        self.num_download_attempts = num_download_attempts
        self.max_in_memory_upload_chunks = max_in_memory_upload_chunks
        self.max_in_memory_download_chunks = max_in_memory_download_chunks
        self.max_bandwidth = max_bandwidth # <--------------------------------------------
        self.use_threads = use_threads
        self._validate_attrs_are_nonzero()

    def _validate_attrs_are_nonzero(self):
        for attr, attr_val, in self.__dict__.items():
            if attr_val is not None and attr_val <= 0:
                raise ValueError(
                    'Provided parameter %s of value %s must be greater than '
                    '0.' % (attr, attr_val))

It’s more LOC initially for the class, but then it’s just adding the Config key to the client’s upload method call and that’s it.

Usage:

upload_config = MyTransferConfig(max_bandwidth=1024**2) # 1MiB/s

s3_client.upload_file(path, bucket_name, object_key, Config=upload_config)

### or

with open(path, "rb") as f:
    s3_client.upload_fileobj(f, bucket_name, object_key, Config=upload_config)

Result of uploading ~40MiB worth of files with a 1 MiB/s bandwidth limit:

I noticed that for low bandwidths like 100 KiB/s, the upload speeds are quite “bursty”:

I could stabilize this behaviour by changing the default value of parameter bytes_threshold of class BandwidthLimitedStream in bandwidth.py from the default 256 * 1024 to a lower value. e.g. 8 * 1024. However, I didn’t investigate the side-effects of doing so, so feel free to propose a better solution. I don’t know if this for example causes any serious overheads on disk I/O or network.

I also wrote a small method to apply this patch automatically:

def patch_s3transfer_upload_chunksize():
    import s3transfer
    bandwidth_module_file = os.path.dirname(s3transfer.__file__) + "/bandwidth.py"
    patch_cmd = "sed -i 's/bytes_threshold=256 \* 1024/bytes_threshold=8 \* 1024/g' " + bandwidth_module_file
    os.system(patch_cmd)

[tong-bluehill]

Since the max_bandwidth is supported in boto/s3transfer but not in boto/boto3, the workaround is using s3transfer to upload with a limit. Paste my code here.

import boto3
s3_client=boto3.client('s3')

import s3transfer
from s3transfer.manager import TransferConfig, TransferManager
tc = TransferConfig(max_bandwidth=100*1000)
tm = TransferManager(s3_client, tc)
tf = tm.upload('/my/file', 'mys3bucket', 'mys3key')
print(tf.result())