S3 Multipart upload using threads

Hi there, It would be great to have an ability to upload chunks to s3 with ThreadPoolExecutor, in other words to send each chunk in a separate thread to make uploading faster. This is how we do in my current project. I think it’s better to include this functionality to smart_open. wdyt? let me know if it’s interesting and I’ll prepare a pull request.

[piskvorky]

But threads won’t help you in I/O bounds operations. They add more CPU, which is not the bottleneck (I think).

I appreciate your offer, but as part of your PR, we’d need to see convincing benchmarks first. Can you post some concrete numbers?

Threads always add a lot of complexity and maintenance and support headache, so unless the gains are obvious, I’m -1 on complicating the code.

[mpenkov]

@vryazanov Thank you for your interest in smart_open. Personally, I don’t think it’s worth including this functionality in smart_open, for several reasons.

First, there are already tools that handle that use case extremely well (for example, the AWS CLI).

Second, there’s a fair bit of effort to implement multi-threaded upload without breaking the existing abstraction of an upload being a write to a file stream. I’m not sure the benefits are worth the cost.