Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Reusing S3 Connection in Threads

See original GitHub issue

I am attempting an upload of files to S3 using concurrent.futures.ThreadPoolExecutor in AWS Lambda. This is a sample of my code:

from concurrent import futures

def my_lambda(event, context):
    def upload_to_s3(file, key):
        s3.Bucket(MY_BUCKET).upload_file(file, key)

    with futures.ThreadPoolExecutor(max_workers=1000) as executor:
        todo = []
        for f in files:
            future = executor.submit(upload_to_s3, f, key)

        results = []
        for future in futures.as_completed(todo):
            res = future.result()

It seems all the files do get upload to S3, but there doesn’t seem to be any time improvement over a sequential upload. I am also getting these logs in CloudWatch:

Connection pool is full, discarding connection:

Did some research and according to this article it is not possible to reuse the connection to S3 since boto3 isn’t thread-safe.

Would like to know if this is indeed correct and if there is a solution to this?

Possibly related: #1128

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:2
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

boarikcommented, Jul 4, 2019

@JordonPhillips Is there some documentation that describes boto3 S3 client thread safety?

I can only find this about the thread safety, and it isn’t explicit about the thread safety.

This link refers to resources rather than clients. Each resource has a meta attribute which holds a boto client under the hood. It sounds to me that the resource holds additional information which makes it thread unsafe, while the client itself might be thread safe. @JordonPhillips Can you please elaborate?

jsyrjalacommented, Sep 20, 2018

@JordonPhillips Is there some documentation that describes boto3 S3 client thread safety?

I can only find this about the thread safety, and it isn’t explicit about the thread safety.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Improve transfer performance of sync ... - Amazon AWS
I'm using the AWS Command Line Interface (AWS CLI) sync command to transfer data on Amazon Simple Storage Service (Amazon S3).
Read more >
AmazonS3 connection management - Stack Overflow
All client classes in the Java SDK are thread safe, so usually it is a better idea to re-use a single client than...
Read more >
Amazon S3 Client :: Quarkiverse Documentation - GitHub Pages
Configuration property Type Default boolean false string localstack AWS SDK client configurations Type Default
Read more >
AWS S3 Connection Pool Error - Google Groups
TransferManager , like all the client classes in the AWS SDK for Java, is thread safe. Call TransferManager.shutdownNow() to release the resources once...
Read more >
AWS S3 with Java - Reactive Support - Baeldung
This means that the same thread keeps going on handling those events – which can originate from any of the active client connections...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found