Reusing S3 Connection in Threads

I am attempting an upload of files to S3 using concurrent.futures.ThreadPoolExecutor in AWS Lambda. This is a sample of my code:

from concurrent import futures


def my_lambda(event, context):
    def upload_to_s3(file, key):
        s3.Bucket(MY_BUCKET).upload_file(file, key)

    with futures.ThreadPoolExecutor(max_workers=1000) as executor:
        todo = []
        for f in files:
            future = executor.submit(upload_to_s3, f, key)
            todo.append(future)

        results = []
        for future in futures.as_completed(todo):
            res = future.result()
            results.append(res)

It seems all the files do get upload to S3, but there doesn’t seem to be any time improvement over a sequential upload. I am also getting these logs in CloudWatch:

Connection pool is full, discarding connection: s3-us-west-2.amazonaws.com

Did some research and according to this article it is not possible to reuse the connection to S3 since boto3 isn’t thread-safe.

Would like to know if this is indeed correct and if there is a solution to this?

Possibly related: #1128

[boarik]

@JordonPhillips Is there some documentation that describes boto3 S3 client thread safety?

I can only find this about the thread safety, and it isn’t explicit about the thread safety. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-multiprocessing

This link refers to resources rather than clients. Each resource has a meta attribute which holds a boto client under the hood. It sounds to me that the resource holds additional information which makes it thread unsafe, while the client itself might be thread safe. @JordonPhillips Can you please elaborate?

[jsyrjala]

@JordonPhillips Is there some documentation that describes boto3 S3 client thread safety?

I can only find this about the thread safety, and it isn’t explicit about the thread safety. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-multiprocessing