Connection pool is full, discarding connection

I’m getting a massive amount of these exceptions while trying to retrieve S3 logs, as I’m retrieving each object in separate threads.

WARNING:botocore.vendored.requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection: bucket.s3.amazonaws.com

Should I be closing connections after using them? Or is this in boto’s hands?

thanks

[JustinTArthur]

This warning is fine. If you dig into the urllib3 connection pool code, it’s basically the pool of persistent connections and not the maximum number of concurrent connections you can have. The connections in the pool are re-used if they haven’t been closed by the endpoint.

If you’d like to boost the size of this pool, it can be done per-endpoint using the low-level botocore config. boto3 example:

import boto3
import botocore

client_config = botocore.config.Config(
    max_pool_connections=25,
)
boto3.client('lambda', config=client_config)

Botocore Config Documentation: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

[myselfdesai]

Solution here … use new feature max_pool_connections s3_client = boto3.client('s3', config=botocore.client.Config(max_pool_connections=50))