Connections to S3 much slower than with boto2?

I have developed a web application with boto (v2.36.0) and am trying to migrate it to use boto3 (v1.1.3). Because the application is deployed on a multi-threaded server, I connect to S3 for each HTTP request/response interaction.

Following the guidance here, I am generating a session per request to ensure thread-safety. However, I have noticed that whereas boto2 takes approximately 0.2 ms to connect to S3, boto3 takes approximately 25 ms.

Here is an IPython session that demonstrates the issue. This was run in IPython 3.0.0 on Python 3.4.3 (Anaconda 2.2.0, 64-bit) on Ubuntu 14.04 LTS on an AWS EC2 instance in us-east-1; however, I have noted similar behavior in Python 2.7 on my local machine.

import boto as boto2
import boto3

access_key_id = ...
secret_access_key = ...
bucket_name = ...
contents = chr(0)

def connect_with_boto2():
    connection = boto2.connect_s3(access_key_id, secret_access_key)
    return connection

def connect_with_boto3():
    session = boto3.session.Session(
        aws_access_key_id=access_key_id, 
        aws_secret_access_key=secret_access_key,
    )
    connection = session.resource('s3')
    return connection

def set_with_boto3():
    connection = connect_with_boto3()
    bucket = connection.Bucket(bucket_name)
    bucket.Object('boto3').put(Body=contents)

def set_with_boto2():
    connection = connect_with_boto2()
    bucket = connection.get_bucket(bucket_name)
    key = boto2.s3.key.Key(bucket, 'boto2')
    key.set_contents_from_string(contents)

%timeit set_with_boto2()
#10 loops, best of 3: 48.9 ms per loop

%timeit set_with_boto3()
#10 loops, best of 3: 73.1 ms per loop

%timeit connect_with_boto2()
#1000 loops, best of 3: 203 µs per loop

%timeit connect_with_boto3()
#10 loops, best of 3: 26.7 ms per loop

Am I setting up these connections correctly, or am I comparing apples-to-oranges? If the latter, is there a way to get the boto3 performance to approximate boto2?

[synhershko]

@kyleknap I believe there are a few more items to consider here

1. Multithreading doesn’t necessarily imply long lived threads. Use of multithreading for parallel download of files for S3 using eventlet for example is common in Python, and for that use case it actually makes sense to create the session and resource once and reuse it from the short lived threads.
2. In addition to 1 above, but not only for that use case, there are many resources that are initialised and could be reused. I.e. the urllib3 error that is thrown often when trying to parallelise S3 reads with boto3 - “Connection pool is full, discarding connection” which points to the fact that multi-threaded usage of a shared session could be beneficial for reasons of sharing internal connection pools (see http://stackoverflow.com/a/18845952/135701, https://github.com/openstack/python-swiftclient/commit/19d7e1812a99d73785146667ae2f3a7156f06898 for example of possible solutions).

Therefore, recommending to create more instances on all multi-threaded use cases isn’t a good advice. It’s probably better to make Session and Resource immutable and threadsafe, and let them manage connection pool sizing etc and by that better support all multithreaded scenarios - including short lived threads used for parallelisation of I/O, and get a good performance gain while at it.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.